Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs!: update pages and skill to cover the recent changes

Karim shoaire45311e2c104
diff --git a/README.md b/README.md
index 0b4f805..a4488f8 100644
--- a/README.md
+++ b/README.md
@@ -291,7 +291,7 @@ MySpider().start()
- **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
- **Remote Browsers**: Instead of launching a browser locally, connect to one that's already running through CDP with `cdp_url`, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with `executable_path`.
-- **Background API Capture**: Pass a URL pattern to `capture_xhr`, and all matching XHR/fetch responses the page makes while loading are collected for you as `Response` objects in `response.captured_xhr` - grab a site's API data without reverse-engineering the requests yourself.
+- **Background API Capture**: Enable `record_requests=True` on a browser session, then search its saved responses with `session.network.search(limit=None)`. Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
### Adaptive Scraping
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index f390b37..f644b47 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 002a928..9469edd 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -402,13 +402,16 @@ async with AsyncStealthySession(max_pages=2) as session:
results = await asyncio.gather(*tasks)
print(session.get_pool_stats())
-# Capture XHR/fetch API calls during page load
-async with AsyncDynamicSession(capture_xhr=r"https://api\.example\.com/.*") as session:
- page = await session.fetch('https://example.com')
- for xhr in page.captured_xhr: # Each is a full Response object
- print(xhr.url, xhr.status, xhr.body)
+async with AsyncDynamicSession(record_requests=True) as session:
+ await session.fetch('https://example.com', network_idle=True)
+
+for response in session.network.search(url_pattern=r"https://api\.example\.com/.*", limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ print(response.meta.get('body_note') or response.body)
```
+Recorded responses belong to the whole browser session. `search()` and `get()` return `Response` objects directly, with `network_id`, `resource_type`, and `request_body` in `.meta`. Searches filter saved history and return 100 matches by default; `limit=None` returns all retained matches. Only supported text response bodies are saved. Saved reads remain local after closure; unfinished captures are not awaited. See [Migrating in v5](references/fetching/dynamic.md#migrating-in-v5) for the removed API capture interface.
+
## References
You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
- `references/mcp-server.md` - MCP server tools, persistent session management, remote browsers over CDP, authentication, and capabilities
diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
index d3fdf98..e6e636b 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
@@ -71,9 +71,8 @@ page.history # Response history of redirections, if any
page.body # Raw response body as bytes
page.encoding # Response encoding
page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
-page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
```
-All fetchers return the `Response` object.
+All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](dynamic.md#network-history) for saved API responses and the v5 migration.
The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:
```python
diff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
index 438bf58..7385aa0 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
@@ -85,7 +85,6 @@ All arguments for `DynamicFetcher` and its session classes:
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
@@ -283,24 +282,6 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
-### Capturing XHR/Fetch Requests
-
-Many SPAs load data through background API calls (XHR/fetch). You can capture these requests by passing a regex URL pattern to `capture_xhr` at the session level:
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True) as session:
- page = session.fetch('https://example.com')
-
- # Access captured XHR responses
- for xhr in page.captured_xhr:
- print(xhr.url, xhr.status)
- print(xhr.body) # Raw response body as bytes
-```
-
-Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.
-
### Network History
Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
@@ -309,26 +290,28 @@ Enable `record_requests` on a browser session to inspect completed requests acro
from scrapling.fetchers import DynamicSession
with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com')
+ session.fetch('https://example.com', network_idle=True)
-for entry in session.network.search(resource_type='document'):
- print(entry.id, entry.url, entry.status)
- if note := entry.response.meta.get('body_note'):
+for response in session.network.search(resource_type='document', limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ if note := response.meta.get('body_note'):
print(note)
else:
- print(entry.response.body)
- print(entry.response.css('title::text').get())
+ print(response.body)
+ print(response.css('title::text').get())
session.network.clear()
```
The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns matching `NetworkRequest` entries in ID order. `url_pattern` is a regular expression; `after_id` selects requests with a larger ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
-- `session.network.get(request_id)` returns one retained entry, or `None` if it is no longer available.
+- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
+- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-Each entry stores `id`, `url`, `method`, `resource_type`, `status`, `request_headers`, `request_body`, and `response_headers`. The request body is bytes or `None`; headers are dictionaries and can be partial if full-header reads fail. The saved Scrapling `response` has the usual `.body` bytes, `.headers`, `.request_headers`, `.json()`, and CSS/XPath methods. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
+Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
+
+To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
@@ -342,7 +325,9 @@ Closing a session does not wait for captures. Captures still running when the pa
By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-Ordinary fetch responses and `capture_xhr` are unchanged; matching XHR/fetch responses still appear in `response.captured_xhr`. Use network history to search completed traffic across the session, including HTTP errors and redirects.
+#### Migrating in v5
+
+`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
### Some Stealth Features
diff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
index b8908e8..f26004e 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
@@ -65,7 +65,6 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
@@ -265,22 +264,22 @@ Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to kee
from scrapling.fetchers import StealthySession
with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com')
+ session.fetch('https://example.com', network_idle=True)
-for entry in session.network.search(resource_type='document'):
- if note := entry.response.meta.get('body_note'):
+for response in session.network.search(resource_type='document', limit=None):
+ if note := response.meta.get('body_note'):
print(note)
else:
- print(entry.response.body)
+ print(response.body)
```
-Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
+Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. `search()` and `get()` return saved `Response` objects directly. Their `.meta` contains `network_id`, `resource_type`, and `request_body`; read `.body`, `.headers`, `.request_headers`, `.status`, and `.url` on the response itself. Search returns at most 100 matches by default; `limit=None` returns all retained matches. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. Ordinary fetch responses and `capture_xhr` behavior are unchanged.
+Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. See [Migrating in v5](dynamic.md#migrating-in-v5) for the API capture changes.
### Session Benefits
diff --git a/agent-skill/Scrapling-Skill/references/spiders/sessions.md b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
index 8d2c344..3428b8d 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/sessions.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
@@ -68,11 +68,12 @@ class ProductSpider(Spider):
# Fast HTTP for listing pages (default)
manager.add("http", FetcherSession())
- # Stealth browser for protected product pages
- manager.add("stealth", AsyncStealthySession(
+ self.browser_session = AsyncStealthySession(
headless=True,
network_idle=True,
- ))
+ record_requests=True,
+ )
+ manager.add("stealth", self.browser_session)
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
@@ -88,8 +89,16 @@ class ProductSpider(Spider):
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
+
+ async def on_close(self):
+ for api_response in self.browser_session.network.search(
+ url_pattern=r"https://api\.shop\.example\.com/.*", limit=None
+ ):
+ self.logger.info(f"Saved API response: {api_response.url} ({api_response.status})")
```
+The example keeps the browser instance and reads its retained API history once in `on_close()`, before the sessions close. The history covers all product pages and stays readable after closure. Pending captures are not awaited, and retention limits still apply.
+
The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
Sessions can also be different instances of the same class with different configurations:
diff --git a/docs/README_AR.md b/docs/README_AR.md
index 9bd3f5a..5e79e0a 100644
--- a/docs/README_AR.md
+++ b/docs/README_AR.md
@@ -279,7 +279,7 @@ MySpider().start()
- **حظر النطاقات والإعلانات**: حظر الطلبات إلى نطاقات محددة (ونطاقاتها الفرعية) أو تفعيل حظر الإعلانات المدمج (~3,500 نطاق إعلانات/تتبع معروف) في الجوالب المعتمدة على المتصفح.
- **منع تسرب DNS**: دعم اختياري لـ DNS-over-HTTPS لتوجيه استعلامات DNS عبر Cloudflare DoH، مما يمنع تسرب DNS عند استخدام Proxy.
- **متصفحات بعيدة**: بدلاً من تشغيل متصفح محلياً، اتصل بمتصفح يعمل بالفعل عبر CDP باستخدام `cdp_url`، سواء كان على نفس الجهاز أو على خادم آخر أو لدى مزود متصفحات مُدار. يمكنك أيضاً توجيه أي جالب معتمد على المتصفح إلى نسخة Chromium الخاصة بك عبر `executable_path`.
-- **التقاط طلبات API في الخلفية**: مرر نمط URL إلى `capture_xhr`، وسيتم جمع كل استجابات XHR/fetch المطابقة التي تصدرها الصفحة أثناء التحميل ككائنات `Response` في `response.captured_xhr` - احصل على بيانات API الموقع دون الحاجة لتحليل الطلبات بنفسك.
+- **التقاط طلبات API في الخلفية**: فعّل `record_requests=True` في جلسة المتصفح، ثم ابحث في الاستجابات المحفوظة باستخدام `session.network.search(limit=None)`. اقرأ بيانات API وأخطاء HTTP من الصفحات التي زرتها دون إرسال الطلبات مرة أخرى. السجل محدود الحجم ويحفظ أجسام الاستجابات النصية.
- **دعم Async**: دعم async كامل عبر جميع الجوالب وفئات الجلسات async المخصصة.
### الاستخراج التكيفي
diff --git a/docs/README_CN.md b/docs/README_CN.md
index ce56a8b..e58bbd1 100644
--- a/docs/README_CN.md
+++ b/docs/README_CN.md
@@ -279,7 +279,7 @@ MySpider().start()
- **域名和广告屏蔽**:在基于浏览器的 Fetcher 中屏蔽对特定域名(及其子域名)的请求,或启用内置广告屏蔽(约 3,500 个已知广告/追踪域名)。
- **DNS 泄漏防护**:可选的 DNS-over-HTTPS 支持,通过 Cloudflare 的 DoH 路由 DNS 查询,防止使用代理时的 DNS 泄漏。
- **远程浏览器**:无需在本地启动浏览器,通过 `cdp_url` 用 CDP 连接到已在运行的浏览器,无论它在同一台机器、另一台主机,还是托管的浏览器服务商。您也可以通过 `executable_path` 让任意浏览器 Fetcher 使用您自己的 Chromium 构建版本。
-- **后台 API 捕获**:向 `capture_xhr` 传入 URL 模式,页面加载过程中所有匹配的 XHR/fetch 响应都会作为 `Response` 对象收集到 `response.captured_xhr` 中--无需自己逆向分析请求即可获取网站的 API 数据。
+- **后台 API 捕获**:在浏览器会话中启用 `record_requests=True`,然后使用 `session.network.search(limit=None)` 搜索已保存的响应。无需再次发送请求,即可读取各个页面加载过程中产生的 API 数据和 HTTP 错误。历史记录有容量限制,并保存文本响应体。
- **Async 支持**:所有 Fetcher 和专用 async Session 类的完整 async 支持。
### 自适应抓取
diff --git a/docs/README_DE.md b/docs/README_DE.md
index 727d2d3..3eee138 100644
--- a/docs/README_DE.md
+++ b/docs/README_DE.md
@@ -279,7 +279,7 @@ MySpider().start()
- **Domain- & Werbeblockierung**: Anfragen an bestimmte Domains (und deren Subdomains) blockieren oder die integrierte Werbeblockierung (~3.500 bekannte Werbe-/Tracker-Domains) in browserbasierten Fetchern aktivieren.
- **DNS-Leak-Prävention**: Optionale DNS-over-HTTPS-Unterstützung zur Weiterleitung von DNS-Anfragen über Cloudflares DoH, um DNS-Leaks bei der Verwendung von Proxys zu verhindern.
- **Remote-Browser**: Statt lokal einen Browser zu starten, verbinden Sie sich per CDP mit `cdp_url` zu einem bereits laufenden Browser -- auf demselben Rechner, einem anderen Host oder bei einem Managed-Browser-Anbieter. Mit `executable_path` können Sie jeden Browser-Fetcher außerdem auf Ihren eigenen Chromium-Build verweisen.
-- **Hintergrund-API-Erfassung**: Übergeben Sie ein URL-Muster an `capture_xhr`, und alle passenden XHR-/Fetch-Antworten, die die Seite beim Laden erzeugt, werden als `Response`-Objekte in `response.captured_xhr` gesammelt -- so kommen Sie an die API-Daten einer Website, ohne die Requests selbst zu rekonstruieren.
+- **Hintergrund-API-Erfassung**: Aktivieren Sie `record_requests=True` in einer Browsersitzung und durchsuchen Sie die gespeicherten Antworten mit `session.network.search(limit=None)`. Lesen Sie API-Daten und HTTP-Fehler über mehrere Seitenaufrufe hinweg, ohne Anfragen erneut zu senden. Der Verlauf ist begrenzt und speichert Text-Antwortkörper.
- **Async-Unterstützung**: Vollständige async-Unterstützung über alle Fetcher und dedizierte async Session-Klassen hinweg.
### Adaptives Scraping
diff --git a/docs/README_ES.md b/docs/README_ES.md
index 34863d7..4722ffd 100644
--- a/docs/README_ES.md
+++ b/docs/README_ES.md
@@ -279,7 +279,7 @@ MySpider().start()
- **Bloqueo de Dominios y Anuncios**: Bloquea solicitudes a dominios específicos (y sus subdominios) o activa el bloqueo de anuncios integrado (~3,500 dominios de anuncios/rastreadores conocidos) en fetchers basados en navegador.
- **Prevención de Fugas DNS**: Soporte opcional de DNS-over-HTTPS para enrutar consultas DNS a través del DoH de Cloudflare, previniendo fugas DNS al usar proxies.
- **Navegadores Remotos**: En lugar de lanzar un navegador localmente, conéctate mediante CDP con `cdp_url` a uno que ya esté en ejecución, ya sea en la misma máquina, en otro host o en un proveedor de navegadores gestionados. También puedes apuntar cualquier fetcher de navegador a tu propia compilación de Chromium con `executable_path`.
-- **Captura de API en Segundo Plano**: Pasa un patrón de URL a `capture_xhr` y todas las respuestas XHR/fetch coincidentes que haga la página durante la carga se recopilarán como objetos `Response` en `response.captured_xhr` - obtén los datos de la API de un sitio sin tener que aplicar ingeniería inversa a las peticiones.
+- **Captura de API en Segundo Plano**: Activa `record_requests=True` en una sesión del navegador y busca las respuestas guardadas con `session.network.search(limit=None)`. Lee datos de API y errores HTTP entre cargas de páginas sin volver a enviar las peticiones. El historial tiene límites y guarda cuerpos de respuesta de texto.
- **Soporte Async**: Soporte async completo en todos los fetchers y clases de sesión async dedicadas.
### Scraping Adaptativo
diff --git a/docs/README_FR.md b/docs/README_FR.md
index 0c7ae67..17f80d5 100644
--- a/docs/README_FR.md
+++ b/docs/README_FR.md
@@ -279,7 +279,7 @@ MySpider().start()
- **Blocage de domaines et publicités** : Bloquez les requêtes vers des domaines spécifiques (et leurs sous-domaines) ou activez le blocage de publicités intégré (~3 500 domaines publicitaires/traceurs connus) dans les fetchers basés sur navigateur.
- **Prévention des fuites DNS** : Support optionnel de DNS-over-HTTPS pour router les requêtes DNS via le DoH de Cloudflare, empêchant les fuites DNS lors de l'utilisation de proxies.
- **Navigateurs distants** : Au lieu de lancer un navigateur localement, connectez-vous via CDP avec `cdp_url` à un navigateur déjà en cours d'exécution, qu'il soit sur la même machine, sur un autre hôte ou chez un fournisseur de navigateurs managés. Vous pouvez aussi pointer n'importe quel fetcher navigateur vers votre propre build de Chromium avec `executable_path`.
-- **Capture des API en arrière-plan** : Passez un motif d'URL à `capture_xhr`, et toutes les réponses XHR/fetch correspondantes émises par la page pendant le chargement sont collectées sous forme d'objets `Response` dans `response.captured_xhr` - récupérez les données de l'API d'un site sans avoir à rétro-concevoir les requêtes vous-même.
+- **Capture des API en arrière-plan** : Activez `record_requests=True` dans une session de navigateur, puis recherchez les réponses enregistrées avec `session.network.search(limit=None)`. Lisez les données API et les erreurs HTTP entre les chargements de pages sans renvoyer les requêtes. L'historique est limité et conserve les corps de réponse textuels.
- **Support async** : Support async complet sur tous les fetchers et classes de sessions async dédiées.
### Scraping adaptatif
diff --git a/docs/README_JP.md b/docs/README_JP.md
index 39398c9..a2f759f 100644
--- a/docs/README_JP.md
+++ b/docs/README_JP.md
@@ -279,7 +279,7 @@ MySpider().start()
- **ドメイン&広告ブロック**:ブラウザベースの Fetcher で特定のドメイン(およびそのサブドメイン)へのリクエストをブロック、または内蔵広告ブロック(約3,500の既知の広告/トラッカードメイン)を有効化。
- **DNS リーク防止**:Proxy 使用時の DNS リークを防ぐため、Cloudflare の DoH 経由で DNS クエリをルーティングするオプションの DNS-over-HTTPS サポート。
- **リモートブラウザ**:ローカルでブラウザを起動する代わりに、`cdp_url` を使って CDP 経由で既に起動しているブラウザに接続できます。同じマシン上でも、別のホストでも、マネージドブラウザサービスでも構いません。`executable_path` を使えば、任意のブラウザ Fetcher を独自ビルドの Chromium に向けることもできます。
-- **バックグラウンド API キャプチャ**:`capture_xhr` に URL パターンを渡すと、読み込み中にページが発行した該当の XHR/fetch レスポンスがすべて `Response` オブジェクトとして `response.captured_xhr` に収集されます。リクエストを自分でリバースエンジニアリングすることなく、サイトの API データを取得できます。
+- **バックグラウンド API キャプチャ**:ブラウザーセッションで `record_requests=True` を有効にし、`session.network.search(limit=None)` で保存済みのレスポンスを検索できます。リクエストを再送信せずに、複数のページ読み込みで発生した API データや HTTP エラーを確認できます。履歴には保存上限があり、テキストのレスポンス本文を保存します。
- **async サポート**:すべての Fetcher および専用 async Session クラス全体での完全な async サポート。
### 適応型スクレイピング
diff --git a/docs/README_KR.md b/docs/README_KR.md
index 7c2191f..51ab3a2 100644
--- a/docs/README_KR.md
+++ b/docs/README_KR.md
@@ -279,7 +279,7 @@ MySpider().start()
- **도메인 및 광고 차단**: 브라우저 기반 Fetcher에서 특정 도메인(및 하위 도메인)으로의 요청을 차단하거나 내장 광고 차단(약 3,500개의 알려진 광고/트래커 도메인)을 활성화합니다.
- **DNS 유출 방지**: 프록시 사용 시 DNS 유출을 방지하기 위해 Cloudflare DoH를 통해 DNS 쿼리를 라우팅하는 선택적 DNS-over-HTTPS 지원.
- **원격 브라우저**: 로컬에서 브라우저를 실행하는 대신 `cdp_url`로 CDP를 통해 이미 실행 중인 브라우저에 연결합니다. 같은 컴퓨터든 다른 호스트든, 관리형 브라우저 서비스든 상관없습니다. `executable_path`로 브라우저 기반 Fetcher가 직접 빌드한 Chromium을 사용하도록 지정할 수도 있습니다.
-- **백그라운드 API 캡처**: `capture_xhr`에 URL 패턴을 전달하면, 페이지가 로드되는 동안 발생한 일치하는 XHR/fetch 응답이 모두 `Response` 객체로 `response.captured_xhr`에 수집됩니다. 요청을 직접 리버스 엔지니어링하지 않고도 사이트의 API 데이터를 얻을 수 있습니다.
+- **백그라운드 API 캡처**: 브라우저 세션에서 `record_requests=True`를 켜고 `session.network.search(limit=None)`으로 저장된 응답을 검색하세요. 요청을 다시 보내지 않고 여러 페이지를 로드하며 발생한 API 데이터와 HTTP 오류를 확인할 수 있습니다. 기록에는 저장 한도가 있으며 텍스트 응답 본문을 저장합니다.
- **비동기 지원**: 모든 Fetcher와 전용 비동기 세션 클래스에서 완전한 비동기를 지원합니다.
### 적응형 스크레이핑
diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.md
index 66acac0..16bb394 100644
--- a/docs/README_PT_BR.md
+++ b/docs/README_PT_BR.md
@@ -281,7 +281,7 @@ MySpider().start()
- **Bloqueio de Domínios e Anúncios**: Bloqueie requisições para domínios específicos (e seus subdomínios) ou habilite o bloqueio nativo de anúncios (~3.500 domínios conhecidos de anúncios/rastreadores) nos fetchers baseados em navegador.
- **Prevenção de Vazamento de DNS**: Suporte opcional a DNS-over-HTTPS para rotear consultas DNS através do DoH da Cloudflare, evitando vazamentos de DNS ao usar proxies.
- **Navegadores Remotos**: Em vez de iniciar um navegador localmente, conecte-se via CDP com `cdp_url` a um que já esteja em execução, seja na mesma máquina, em outro host ou em um provedor de navegadores gerenciados. Você também pode apontar qualquer fetcher de navegador para a sua própria build do Chromium com `executable_path`.
-- **Captura de API em Segundo Plano**: Passe um padrão de URL para `capture_xhr` e todas as respostas XHR/fetch correspondentes que a página fizer durante o carregamento serão coletadas como objetos `Response` em `response.captured_xhr` - obtenha os dados da API de um site sem precisar fazer engenharia reversa das requisições.
+- **Captura de API em Segundo Plano**: Ative `record_requests=True` em uma sessão do navegador e busque as respostas salvas com `session.network.search(limit=None)`. Leia dados de API e erros HTTP entre carregamentos de páginas sem enviar as requisições novamente. O histórico tem limites e salva corpos de resposta de texto.
- **Suporte Async**: Suporte assíncrono completo em todos os fetchers e classes dedicadas de sessão async.
### Scraping Adaptativo
diff --git a/docs/README_RU.md b/docs/README_RU.md
index 41287a1..9236d33 100644
--- a/docs/README_RU.md
+++ b/docs/README_RU.md
@@ -282,7 +282,7 @@ MySpider().start()
- **Блокировка доменов и рекламы**: Блокируйте запросы к определённым доменам (и их поддоменам) или включите встроенную блокировку рекламы (~3 500 известных рекламных/трекерных доменов) в браузерных Fetcher'ах.
- **Защита от утечки DNS**: Опциональная поддержка DNS-over-HTTPS для маршрутизации DNS-запросов через Cloudflare DoH, предотвращая утечку DNS при использовании прокси.
- **Удалённые браузеры**: Вместо запуска браузера локально подключайтесь по CDP через `cdp_url` к уже запущенному браузеру - на той же машине, на другом хосте или у провайдера управляемых браузеров. С помощью `executable_path` можно также направить любой браузерный Fetcher на вашу собственную сборку Chromium.
-- **Перехват фоновых API-запросов**: Передайте шаблон URL в `capture_xhr`, и все подходящие XHR/fetch-ответы, которые страница делает во время загрузки, будут собраны как объекты `Response` в `response.captured_xhr` - получайте данные API сайта, не разбирая запросы вручную.
+- **Перехват фоновых API-запросов**: Включите `record_requests=True` в сессии браузера и ищите сохранённые ответы с помощью `session.network.search(limit=None)`. Читайте данные API и ошибки HTTP после загрузки разных страниц без повторной отправки запросов. История ограничена по размеру и сохраняет текстовые тела ответов.
- **Поддержка async**: Полная async-поддержка во всех Fetcher'ах и выделенных async-классах сессий.
### Адаптивный скрапинг
diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md
index 483aa5b..33f339b 100644
--- a/docs/fetching/choosing.md
+++ b/docs/fetching/choosing.md
@@ -77,9 +77,8 @@ page.history # Response history of redirections, if any
page.body # Raw response body as bytes
page.encoding # Response encoding
page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
-page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
```
-All fetchers return the `Response` object.
+All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](dynamic.md#network-history) for saved API responses and the v5 migration.
The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:
```python
diff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.md
index d51f333..112d29a 100644
--- a/docs/fetching/dynamic.md
+++ b/docs/fetching/dynamic.md
@@ -97,7 +97,6 @@ Scrapling provides many options with this fetcher and its session classes. To ma
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
@@ -337,26 +336,6 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
-### Capturing XHR/Fetch Requests
-
-Many SPAs load data through background API calls (XHR/fetch). You can capture these requests by passing a regex URL pattern to `capture_xhr` at the session level:
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True) as session:
- page = session.fetch('https://example.com')
-
- # Access captured XHR responses
- for xhr in page.captured_xhr:
- print(xhr.url, xhr.status)
- print(xhr.body) # Raw response body as bytes
-```
-
-Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.
-
-Use `capture_xhr=".*"` to capture all XHR/fetch URLs. Invalid regular expressions raise an error before the browser starts.
-
### Network History
Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
@@ -365,26 +344,28 @@ Enable `record_requests` on a browser session to inspect completed requests acro
from scrapling.fetchers import DynamicSession
with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com')
+ session.fetch('https://example.com', network_idle=True)
-for entry in session.network.search(resource_type='document'):
- print(entry.id, entry.url, entry.status)
- if note := entry.response.meta.get('body_note'):
+for response in session.network.search(resource_type='document', limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ if note := response.meta.get('body_note'):
print(note)
else:
- print(entry.response.body)
- print(entry.response.css('title::text').get())
+ print(response.body)
+ print(response.css('title::text').get())
session.network.clear()
```
The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns matching `NetworkRequest` entries in ID order. `url_pattern` is a regular expression; `after_id` selects requests with a larger ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
-- `session.network.get(request_id)` returns one retained entry, or `None` if it is no longer available.
+- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
+- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-Each entry stores `id`, `url`, `method`, `resource_type`, `status`, `request_headers`, `request_body`, and `response_headers`. The request body is bytes or `None`; headers are dictionaries and can be partial if full-header reads fail. The saved Scrapling `response` has the usual `.body` bytes, `.headers`, `.request_headers`, `.json()`, and CSS/XPath methods. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
+Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
+
+To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
@@ -398,7 +379,9 @@ Closing a session does not wait for captures. Captures still running when the pa
By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-Ordinary fetch responses and `capture_xhr` are unchanged; matching XHR/fetch responses still appear in `response.captured_xhr`. Use network history to search completed traffic across the session, including HTTP errors and redirects.
+#### Migrating in v5
+
+`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
### Some Stealth Features
diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md
index 48fb847..2d57fbf 100644
--- a/docs/fetching/stealthy.md
+++ b/docs/fetching/stealthy.md
@@ -76,7 +76,6 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
@@ -85,7 +84,7 @@ In session classes, all these arguments can be set globally for the session. Sti
!!! note "Notes:"
- 1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`. The `capture_xhr` argument is shared with `DynamicFetcher`.
+ 1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
2. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
@@ -276,22 +275,22 @@ Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to kee
from scrapling.fetchers import StealthySession
with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com')
+ session.fetch('https://example.com', network_idle=True)
-for entry in session.network.search(resource_type='document'):
- if note := entry.response.meta.get('body_note'):
+for response in session.network.search(resource_type='document', limit=None):
+ if note := response.meta.get('body_note'):
print(note)
else:
- print(entry.response.body)
+ print(response.body)
```
-Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
+Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. `search()` and `get()` return saved `Response` objects directly. Their `.meta` contains `network_id`, `resource_type`, and `request_body`; read `.body`, `.headers`, `.request_headers`, `.status`, and `.url` on the response itself. Search returns at most 100 matches by default; `limit=None` returns all retained matches. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. Ordinary fetch responses and `capture_xhr` behavior are unchanged.
+Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. See [Migrating in v5](dynamic.md#migrating-in-v5) for the API capture changes.
### Session Benefits
diff --git a/docs/index.md b/docs/index.md
index f0f87ab..76e4d72 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -133,7 +133,7 @@ MySpider().start()
- **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
- **Remote Browsers**: Instead of launching a browser locally, connect to one that's already running through CDP with `cdp_url`, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with `executable_path`.
-- **Background API Capture**: Pass a URL pattern to `capture_xhr`, and all matching XHR/fetch responses the page makes while loading are collected for you as `Response` objects in `response.captured_xhr` - grab a site's API data without reverse-engineering the requests yourself.
+- **Background API Capture**: Enable `record_requests=True` on a browser session, then search its saved responses with `session.network.search(limit=None)`. Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
### Adaptive Scraping
diff --git a/docs/spiders/sessions.md b/docs/spiders/sessions.md
index d1b146e..d62d72e 100644
--- a/docs/spiders/sessions.md
+++ b/docs/spiders/sessions.md
@@ -73,13 +73,12 @@ class ProductSpider(Spider):
# Fast HTTP for listing pages (default)
manager.add("http", FetcherSession())
- # Stealth browser for protected product pages
- # capture_xhr captures background API calls matching the regex
- manager.add("stealth", AsyncStealthySession(
+ self.browser_session = AsyncStealthySession(
headless=True,
network_idle=True,
- capture_xhr=r"https://api\.shop\.example\.com/.*",
- ))
+ record_requests=True,
+ )
+ manager.add("stealth", self.browser_session)
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
@@ -91,16 +90,20 @@ class ProductSpider(Spider):
yield response.follow(next_page)
async def parse_product(self, response: Response):
- # Access captured XHR/fetch API calls (if capture_xhr was set on the session)
- for xhr in response.captured_xhr:
- self.logger.info(f"Captured API call: {xhr.url} ({xhr.status})")
-
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
+
+ async def on_close(self):
+ for api_response in self.browser_session.network.search(
+ url_pattern=r"https://api\.shop\.example\.com/.*", limit=None
+ ):
+ self.logger.info(f"Saved API response: {api_response.url} ({api_response.status})")
```
+The example keeps the browser instance and reads its retained API history once in `on_close()`, before the sessions close. The history covers all product pages and stays readable after closure. Pending captures are not awaited, and retention limits still apply.
+
The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
Note that the sessions don't have to be from different classes only, but can be the same session, but different instances with different configurations, for example, like below: