🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: update pages and the skill with the latest changes
Karim shoair00d5f3da6e8e
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 05e655d..f390b37 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.mdindex 6cfc55b..438bf58 100644--- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md+++ b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md@@ -86,6 +86,8 @@ All arguments for `DynamicFetcher` and its session classes:| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ || retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ || capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |+| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |+| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ || executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `blocked_domains`, `proxy`, and `selector_config`.@@ -299,6 +301,49 @@ with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True)Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.+### Network History++Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.++```python+from scrapling.fetchers import DynamicSession++with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:+ session.fetch('https://example.com')++for entry in session.network.search(resource_type='document'):+ print(entry.id, entry.url, entry.status)+ if note := entry.response.meta.get('body_note'):+ print(note)+ else:+ print(entry.response.body)+ print(entry.response.css('title::text').get())++session.network.clear()+```++The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.++- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns matching `NetworkRequest` entries in ID order. `url_pattern` is a regular expression; `after_id` selects requests with a larger ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.+- `session.network.get(request_id)` returns one retained entry, or `None` if it is no longer available.+- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.++Each entry stores `id`, `url`, `method`, `resource_type`, `status`, `request_headers`, `request_body`, and `response_headers`. The request body is bytes or `None`; headers are dictionaries and can be partial if full-header reads fail. The saved Scrapling `response` has the usual `.body` bytes, `.headers`, `.request_headers`, `.json()`, and CSS/XPath methods. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.++The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.++Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.++Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.++The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.++Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.++By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.++Ordinary fetch responses and `capture_xhr` are unchanged; matching XHR/fetch responses still appear in `response.captured_xhr`. Use network history to search completed traffic across the session, including HTTP errors and redirects.+### Some Stealth Features```pythondiff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.mdindex 9a44165..b8908e8 100644--- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md+++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md@@ -66,6 +66,8 @@ Scrapling provides many options with this fetcher and its session classes. Befor| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ || retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ || capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |+| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |+| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ || executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.@@ -255,6 +257,31 @@ Keeping the tabs open also means the page you fetched is still there for the nexVersions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.+### Network History++Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to keep completed requests across page loads, actions, tabs, and temporary proxy contexts. Find API calls, HTTP errors, and redirects, then read saved Scrapling responses without sending requests again. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included. Saved headers and bodies remain readable after the session closes.++```python+from scrapling.fetchers import StealthySession++with StealthySession(record_requests=True, max_recorded_requests=1000) as session:+ session.fetch('https://example.com')++for entry in session.network.search(resource_type='document'):+ if note := entry.response.meta.get('body_note'):+ print(note)+ else:+ print(entry.response.body)+```++Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.++The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.++Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.++Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. Ordinary fetch responses and `capture_xhr` behavior are unchanged.+### Session Benefits- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.mdindex a6e19bb..697ae78 100644--- a/agent-skill/Scrapling-Skill/references/mcp-server.md+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md@@ -1,8 +1,8 @@# Scrapling MCP Server-The Scrapling MCP server exposes fourteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.+The Scrapling MCP server exposes sixteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`browser_fetch_once`, `browser_fetch_many_once`) each launch and close their own browser, while `browser_fetch` and `session_make_request` work through sessions opened with `browser_open`/`open_request_session`.-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text. `browser_evaluate` returns compact JSON in one plain text block.+Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). Bulk tools return a list of these responses. The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON; the request list supports pagination. `browser_evaluate` returns compact JSON in one plain text block.## Shadow DOM@@ -107,6 +107,8 @@ Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_urUse `solve_cloudflare` on `browser_fetch` to solve Cloudflare challenges through the session.+Recording of completed requests starts automatically for this session. Use `browser_network_requests` to find traffic and `browser_network_request` to inspect a recorded request.+### `open_request_session` -- Create a persistent HTTP requests sessionOpens an HTTP session (no browser) that stays alive across multiple `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests. Returns the same `SessionCreatedModel` receipt and shows in `list_sessions` as a `static` session.@@ -231,6 +233,50 @@ The result is compact JSON in one `TextContent` block with `structured_output=FaThe page stays reserved during evaluation and is released after success, failure, or cancellation. Scripts can change page state or make requests. Errors and cancellation do not undo those effects or guarantee that browser-side JavaScript stops. Use `browser_snapshot` or another read to check effects before repeating a script. Unknown or HTTP sessions and missing, closed, or busy pages return an error; JavaScript errors propagate.+### `browser_network_requests` -- Search recorded browser requests++Reads completed requests from a session opened with `browser_open`, across page loads, actions, tabs, and navigation. It does not navigate, reserve a page, or send requests. Recording starts automatically and saves response headers and supported text bodies. Completed HTTP 4xx and 5xx responses are included; failed and unfinished requests are omitted. The history retains up to 1,000 requests, with fixed limits of 1 MiB per response body and 20 MiB combined; old records are removed when retention limits are reached.++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `session_id` | str | required | ID from `browser_open` |+| `url_pattern` | str or null | null | Regular expression to match request URLs |+| `include_static` | bool | false | Include all resource types; otherwise keep XHR/fetch and HTTP errors |+| `after_id` | int | 0 | Nonnegative cursor; return requests with larger IDs |+| `limit` | int | 50 | Maximum returned requests, from 1 to 200 |++Returns structured JSON with `requests`, `next_cursor`, `has_more`, and `dropped_count`. Each request contains integer `id` and `status` fields, plus string `url`, `method`, and `resource_type` fields. `dropped_count` counts older entries removed by the history limits.++Pass `next_cursor` as `after_id` to read later entries. `has_more` means more retained entries currently match the same filters. An empty result has `requests=[]` and `has_more=false`; `next_cursor` is the greater of `after_id` and the latest saved request ID. Use `include_static=true` to include successful document requests and saved redirects. Entries receive increasing IDs when capture finishes. Reads do not wait for captures; use browser actions to wait for the page when needed.++Both network tools provide an MCP output schema and `structured_content`. The SDK also returns the JSON in a text block for clients that read text content.++### `browser_network_request` -- Inspect one recorded request++Reads one recorded request without navigating, reserving a page, or sending it again. All parts use saved data, which remains available after page changes and temporary-context closure. Reads do not call browser methods. An entry appears only after its response has been converted and saved.++The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.++Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `session_id` | str | required | ID from `browser_open` |+| `request_id` | int | required | Positive recorded request ID from `browser_network_requests` |+| `part` | str | `"summary"` | `"summary"`, `"request_headers"`, `"request_body"`, `"response_headers"`, or `"response_body"` |++Returns structured JSON with `request_id`, `part`, `data`, and `note`.++- Summary `data` is an object with `id`, `url`, `method`, `resource_type`, and `status`.+- Header `data` is an object of header names and values. Partial headers keep that object and add a `note`.+- Body `data` is the full saved text for any text resource type, including documents and API calls. JSON response bodies stay strings; they are not parsed into objects.++Saved empty text, HEAD responses, and status codes 204, 205, and 304 return `data=""` and `note=null`. Missing, unreadable, oversized, or non-text body data returns `data=null` and a `note` with the reason. Skipped non-text responses use `Non-text body; not saved.`. A missing or evicted request ID returns an error.++Saved response data survives browser cleanup. Closing a session does not wait for captures; captures still running at navigation or closure may be omitted. The limits cover retained request entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.++Both network tools reject unknown or HTTP sessions. Read needed details before `close_session`, which removes the MCP session.+### `session_make_request` -- HTTP request through an open requests sessionMakes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.@@ -301,6 +347,8 @@ The page stays reserved during capture and is released after success, failure, o| Read the current page's AI ARIA snapshot | `browser_snapshot` with `session_id` || Chain mouse, field, keyboard, or wait actions | `browser_actions` with `session_id` || Custom JavaScript extraction or page tasks | `browser_evaluate` with `session_id` |+| Find API calls, HTTP errors, or redirects | `browser_network_requests` with `session_id` |+| Read recorded request headers or body data | `browser_network_request` with `session_id` and `request_id` |Start with `make_request` (fastest, lowest resource cost). Use `browser_fetch_once` when JavaScript rendering is needed or HTTP requests are blocked. Enable `solve_cloudflare` for Cloudflare challenges. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.@@ -391,7 +439,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com## Connecting to remote browsers-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. The `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, and `browser_screenshot` as usual.+`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. The `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, and `browser_network_request` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.mdindex d5b4b6d..01b87c9 100644--- a/docs/ai/mcp-server.md+++ b/docs/ai/mcp-server.md@@ -6,7 +6,7 @@ The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful W## Features-The Scrapling MCP Server provides fourteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.+The Scrapling MCP Server provides sixteen tools for web scraping and browser interaction, split into two modes: one-shot tools that each launch and close their own browser/client, and session tools that open a browser or an HTTP session once and then work through it.### One-shot tools@@ -53,6 +53,15 @@ The Scrapling MCP Server provides fourteen tools for web scraping and browser in- **`browser_evaluate`**: Run JavaScript on the current page for custom data extraction, calculations, and page-specific tasks. The AI can collect data from several elements, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.+#### Network History++- **`browser_network_requests`**: Find completed requests across tabs, page loads, and browser actions. Discover the API calls behind a page's data, inspect HTTP errors, and follow saved redirects. Static resources are hidden by default to keep results small. Returns request details as JSON, with pagination for long histories.+- **`browser_network_request`**: Inspect one recorded request's summary, headers, sent data, or text response body as structured JSON. Returns the selected part in full without loading the page or sending the request again. Saved responses remain available after page changes. Size limits keep the history bounded; a note explains non-text, oversized, or unreadable bodies. Read needed details before closing the MCP session.++Response bodies are read and saved only for supported text `Content-Type` values, such as HTML, JSON, XML, or JavaScript. Binary, missing, or unrecognized content types keep request metadata and headers without reading or saving their response bodies. The browser still loads resources normally.++Only completed requests are saved, including HTTP 4xx and 5xx responses. Failed and unfinished requests are omitted. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from Playwright and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback.+### Shadow DOMSet `pierce_shadow=true` on `browser_fetch_once`, `browser_fetch_many_once`, or `browser_fetch` to include open Shadow DOM content. It defaults to `false`. For persistent sessions, pass it on each `browser_fetch` call. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.@@ -212,7 +221,7 @@ You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before sta```Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.```-The `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, and `browser_screenshot` as usual.+The `session_id` you get back is used with `browser_fetch`, `browser_snapshot`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, and `browser_network_request` as usual.The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:```commandlinediff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.mdindex e2cf3e7..6eb229c 100644--- a/docs/api-reference/mcp-server.md+++ b/docs/api-reference/mcp-server.md@@ -5,7 +5,7 @@ search:# MCP Server API Reference-The **Scrapling MCP Server** provides fourteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.+The **Scrapling MCP Server** provides sixteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.You can start the MCP server by running:@@ -30,12 +30,30 @@ server = ScraplingMCPServer(executable_path="/path/to/chromium")## Response Model-The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. `browser_evaluate` returns its result as JSON in a plain text block.+The standardized response structure returned by the fetch and HTTP request tools. Bulk tools return a list of these responses. Session management tools use the models below, `browser_screenshot` returns an image of the current page or one element, plus the current page URL, as image and text content blocks, and `browser_snapshot` and `browser_actions` return plain text. Both network tools return structured JSON using the models below. `browser_evaluate` returns its result as JSON in a plain text block.## ::: scrapling.core.ai.ResponseModelhandler: python:docstring:+## Network Request Models++`browser_network_requests` returns `NetworkRequestsModel` with `requests`, `next_cursor`, `has_more`, and `dropped_count`. Each `NetworkRequestInfo` contains `id`, `url`, `method`, `resource_type`, and `status`. Pass `next_cursor` as `after_id` to continue; `has_more` indicates more current matches.++`browser_network_request` returns `NetworkRequestModel` with `request_id`, `part`, `data`, and `note`. Summaries and headers are objects; the full saved body text is returned as a string, including JSON response text. `note` explains missing body data or partial headers.++## ::: scrapling.core.ai.NetworkRequestInfo+ handler: python+ :docstring:++## ::: scrapling.core.ai.NetworkRequestsModel+ handler: python+ :docstring:++## ::: scrapling.core.ai.NetworkRequestModel+ handler: python+ :docstring:+## Session ModelsModel classes for session management:@@ -66,6 +84,8 @@ The main MCP server class that provides all web scraping tools:Wait for content, loading overlays, page load events, or settled network activity, or add a fixed pause before continuing.- **`browser_evaluate`**: Run JavaScript on the current page to extract custom data, calculate results, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.+- **`browser_network_requests`**: Search completed browser requests across tabs, navigation, and actions. Find API calls, HTTP errors, and saved redirects, with static resources hidden by default for smaller results. Failed and unfinished requests are omitted. Returns structured request details with pagination.+- **`browser_network_request`**: Read a recorded request's summary, headers, sent data, or text response body as structured JSON without repeating the request. Returns the selected part in full, including saved body text. Saved responses remain available after page changes, including HTTP 4xx and 5xx responses. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from Playwright and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback. Only response bodies with a supported text `Content-Type` are read and saved. Binary, missing, or unrecognized types keep metadata and headers without body reads; browser loading is unchanged. Size limits bound the saved history, and an optional note explains non-text, oversized, or unreadable bodies. Read needed details before closing the MCP session.- **`browser_fetch`**: Fetch a page through an open stealthy browser session and extract HTML, Markdown, text, or a structured snapshot. Snapshots can cover the whole page or a chosen element and include element references, positions, and sizes for later interaction.## ::: scrapling.core.ai.ScraplingMCPServerdiff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.mdindex 311f1e9..33f16f9 100644--- a/docs/fetching/dynamic.md+++ b/docs/fetching/dynamic.md@@ -63,42 +63,44 @@ The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed br## Full list of argumentsScrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of how to use most of them.-| Argument | Description | Optional |-|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|-| url | Target url | ❌ |-| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |-| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |-| cookies | Set cookies for the next request. | ✔️ |-| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |-| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |-| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |-| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |-| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |-| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |-| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |-| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |-| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |-| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |-| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |-| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |-| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |-| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |-| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |-| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |-| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |-| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |-| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |-| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |-| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |-| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |-| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |-| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |-| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |-| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |-| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |-| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |+| Argument | Description | Optional |+|:---------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|+| url | Target url | ❌ |+| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |+| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |+| cookies | Set cookies for the next request. | ✔️ |+| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |+| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |+| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |+| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |+| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |+| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |+| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |+| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |+| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |+| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |+| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |+| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |+| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |+| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |+| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |+| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |+| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |+| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |+| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |+| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |+| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |+| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |+| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |+| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |+| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |+| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |+| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |+| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |+| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |+| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |+| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |+| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `blocked_domains`, `proxy`, and `selector_config`.@@ -353,6 +355,49 @@ with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True)Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.+### Network History++Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.++```python+from scrapling.fetchers import DynamicSession++with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:+ session.fetch('https://example.com')++for entry in session.network.search(resource_type='document'):+ print(entry.id, entry.url, entry.status)+ if note := entry.response.meta.get('body_note'):+ print(note)+ else:+ print(entry.response.body)+ print(entry.response.css('title::text').get())++session.network.clear()+```++The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.++- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns matching `NetworkRequest` entries in ID order. `url_pattern` is a regular expression; `after_id` selects requests with a larger ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.+- `session.network.get(request_id)` returns one retained entry, or `None` if it is no longer available.+- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.++Each entry stores `id`, `url`, `method`, `resource_type`, `status`, `request_headers`, `request_body`, and `response_headers`. The request body is bytes or `None`; headers are dictionaries and can be partial if full-header reads fail. The saved Scrapling `response` has the usual `.body` bytes, `.headers`, `.request_headers`, `.json()`, and CSS/XPath methods. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.++The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.++Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.++Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.++The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.++Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.++By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.++Ordinary fetch responses and `capture_xhr` are unchanged; matching XHR/fetch responses still appear in `response.captured_xhr`. Use network history to search completed traffic across the session, including HTTP errors and redirects.+### Some Stealth Features```pythondiff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.mdindex 59149e3..48fb847 100644--- a/docs/fetching/stealthy.md+++ b/docs/fetching/stealthy.md@@ -38,46 +38,48 @@ The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamiScrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments-| Argument | Description | Optional |-|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|-| url | Target url | ❌ |-| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |-| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |-| cookies | Set cookies for the next request. | ✔️ |-| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |-| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |-| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |-| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |-| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |-| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |-| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |-| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |-| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |-| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |-| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |-| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |-| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |-| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |-| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |-| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |-| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |-| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |-| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |-| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |-| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |-| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |-| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |-| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |-| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |-| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |-| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |-| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |-| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |-| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |-| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |-| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |+| Argument | Description | Optional |+|:---------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|+| url | Target url | ❌ |+| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |+| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |+| cookies | Set cookies for the next request. | ✔️ |+| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |+| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |+| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |+| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |+| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |+| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |+| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |+| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |+| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |+| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |+| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |+| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |+| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |+| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |+| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |+| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |+| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |+| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |+| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |+| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |+| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |+| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |+| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |+| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |+| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |+| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |+| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |+| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |+| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |+| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |+| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |+| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |+| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |+| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |+| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |+| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.@@ -266,6 +268,31 @@ Keeping the tabs open also means the page you fetched is still there for the nexVersions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.+### Network History++Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to keep completed requests across page loads, actions, tabs, and temporary proxy contexts. Find API calls, HTTP errors, and redirects, then read saved Scrapling responses without sending requests again. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included. Saved headers and bodies remain readable after the session closes.++```python+from scrapling.fetchers import StealthySession++with StealthySession(record_requests=True, max_recorded_requests=1000) as session:+ session.fetch('https://example.com')++for entry in session.network.search(resource_type='document'):+ if note := entry.response.meta.get('body_note'):+ print(note)+ else:+ print(entry.response.body)+```++Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.++The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.++Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.++Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. Ordinary fetch responses and `capture_xhr` behavior are unchanged.+### Session Benefits- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.