Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: update the docs accordingly with the latest breaking changes

Karim shoair2228316b40a8
diff --git a/README.md b/README.md
index bd492f0..ec81ed7 100644
--- a/README.md
+++ b/README.md
@@ -68,7 +68,7 @@ Its parser learns from website changes and automatically relocates your elements
Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar!
products = p.css('.product', auto_save=True) # Scrape data that survives website design changes!
@@ -255,9 +255,9 @@ MySpider().start()
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
-- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium and Google's Chrome.
+- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `StealthyFetcher` class supporting Playwright's Chromium and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
-- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
+- **Session Management**: Persistent session support with `FetcherSession` and `StealthySession` classes for cookie and state management across requests.
- **Proxy Rotation**: Built-in `ProxyRotator` with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
- **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
@@ -271,7 +271,7 @@ MySpider().start()
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
### AI Features
-- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method), browser fetches, and stealth fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `SiteToMarkdownSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -323,14 +323,14 @@ data = page.css('#padded_content a').getall()
```
Full browser automation
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # XPath selector if you prefer it
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -436,7 +436,7 @@ And it works precisely the same way!
### Async Session Management Examples
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
page1 = session.get('https://quotes.toscrape.com/')
@@ -471,7 +471,7 @@ Extract pages to a file directly without programming (Extracts the content insid
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index d627e1a..681a192 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 2df07c8..17afc10 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -73,8 +73,7 @@ Commands:
post Perform a POST request and save the content to a file.
put Perform a PUT request and save the content to a file.
delete Perform a DELETE request and save the content to a file.
- fetch Use a browser to fetch content with browser automation and flexible options.
- stealthy-fetch Use a stealthy browser to fetch content with advanced stealth features.
+ fetch Use StealthyFetcher for browser automation and anti-bot protection.
```
### Usage pattern
@@ -87,10 +86,9 @@ Commands:
Which command to use generally:
- Use **`get`** with simple websites, blogs, or news articles.
-- Use **`fetch`** with modern web apps, or sites with dynamic content.
-- Use **`stealthy-fetch`** with protected sites, Cloudflare, or anti-bot systems.
+- Use **`fetch`** with JavaScript pages, browser automation, or protected sites, including Cloudflare.
-> When unsure, start with `get`. If it fails or returns empty content, escalate to `fetch`, then `stealthy-fetch`. The speed of `fetch` and `stealthy-fetch` is nearly the same, so you are not sacrificing anything.
+> When unsure, start with `get`. If the page needs JavaScript or blocks HTTP requests, use `fetch`. Enable `--solve-cloudflare` when the page has a Cloudflare challenge.
#### Key options (requests)
@@ -141,7 +139,7 @@ scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Ac
#### Key options (browsers)
-Both (`fetch` / `stealthy-fetch`) share options:
+The `fetch` command accepts these options:
| Option | Input type | Description |
@@ -161,21 +159,11 @@ Both (`fetch` / `stealthy-fetch`) share options:
| --block-ads / --no-block-ads | None | Block requests to ~3,500 known ad and tracker domains (default: False) |
| --executable-path | TEXT | Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set. |
| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False). Also enables ad blocking automatically. |
-
-This option is specific to `fetch` only:
-
-| Option | Input type | Description |
-|:---------|:----------:|:------------------------------------------------------------|
-| --locale | TEXT | Specify user locale. Defaults to the system default locale. |
-
-And these options are specific to `stealthy-fetch` only:
-
-| Option | Input type | Description |
-|:-------------------------------------------|:----------:|:------------------------------------------------|
-| --block-webrtc / --allow-webrtc | None | Block WebRTC entirely (default: False) |
-| --solve-cloudflare / --no-solve-cloudflare | None | Solve Cloudflare challenges (default: False) |
-| --allow-webgl / --block-webgl | None | Allow WebGL (default: True) |
-| --hide-canvas / --show-canvas | None | Add noise to canvas operations (default: False) |
+| --locale | TEXT | Specify user locale. Defaults to the system default locale. |
+| --block-webrtc / --allow-webrtc | None | Force WebRTC to respect proxy settings (default: False). |
+| --solve-cloudflare / --no-solve-cloudflare | None | Solve Cloudflare challenges (default: False). |
+| --allow-webgl / --block-webgl | None | Allow WebGL (default: True). |
+| --hide-canvas / --show-canvas | None | Add noise to canvas operations (default: False). |
Examples:
@@ -191,13 +179,13 @@ scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
# Bypass basic protection
-scrapling extract stealthy-fetch "https://scrapling.requestcatcher.com" content.md
+scrapling extract fetch "https://scrapling.requestcatcher.com" content.md
# Solve Cloudflare challenges
-scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
+scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use a proxy for anonymity.
-scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
+scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
```
@@ -242,29 +230,28 @@ data = page.css('#padded_content a').getall()
```
Full browser automation
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # XPath selector if you prefer it
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
### Shadow DOM
-Use `pierce_shadow=True` with dynamic or stealth Python fetchers and sessions to include open shadow roots. It defaults to `False` and can be overridden per request. For MCP, pass `pierce_shadow=true` to `browser_fetch_once` or each `browser_fetch` call.
+Use `pierce_shadow=True` with StealthyFetcher and its session classes to include open shadow roots. It defaults to `False` and can be overridden per request. For MCP, pass `pierce_shadow=true` to `browser_fetch_once` or each `browser_fetch` call.
-Both browser CLI commands accept `--pierce-shadow` and `--no-pierce-shadow`:
+The browser CLI command accepts `--pierce-shadow` and `--no-pierce-shadow`:
```bash
scrapling extract fetch "https://example.com" content.md --pierce-shadow --ai-targeted
-scrapling extract stealthy-fetch "https://example.com" content.md --pierce-shadow --ai-targeted
```
-The response includes `<shadow-root>` wrappers and `<slot>` elements. Use descendant selectors, or include those elements in direct-child paths. See [Shadow DOM](references/fetching/dynamic.md#shadow-dom) for assigned content, hidden slots, and extraction limits.
+The response includes `<shadow-root>` wrappers and `<slot>` elements. Use descendant selectors, or include those elements in direct-child paths. See [Shadow DOM](references/fetching/stealthy.md#shadow-dom) for assigned content, hidden slots, and extraction limits.
### Spiders
Build full crawlers with concurrent requests, multiple session types, and pause/resume:
@@ -383,7 +370,7 @@ And it works precisely the same way!
### Async Session Management Examples
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
page1 = session.get('https://quotes.toscrape.com/')
@@ -402,7 +389,7 @@ async with AsyncStealthySession(max_pages=2) as session:
results = await asyncio.gather(*tasks)
print(session.get_pool_stats())
-async with AsyncDynamicSession(record_requests=True) as session:
+async with AsyncStealthySession(record_requests=True) as session:
await session.fetch('https://example.com', network_idle=True)
for response in session.network.search(url_pattern=r"https://api\.example\.com/.*", limit=None):
@@ -410,7 +397,7 @@ for response in session.network.search(url_pattern=r"https://api\.example\.com/.
print(response.meta.get('body_note') or response.body)
```
-Recorded responses belong to the whole browser session. `search()` and `get()` return `Response` objects directly, with `network_id`, `resource_type`, and `request_body` in `.meta`. Searches filter saved history and return 100 matches by default; `limit=None` returns all retained matches. Only supported text response bodies are saved. Saved reads remain local after closure; unfinished captures are not awaited. See [Migrating in v5](references/fetching/dynamic.md#migrating-in-v5) for the removed API capture interface.
+Recorded responses belong to the whole browser session. `search()` and `get()` return `Response` objects directly, with `network_id`, `resource_type`, and `request_body` in `.meta`. Searches filter saved history and return 100 matches by default; `limit=None` returns all retained matches. Only supported text response bodies are saved. Saved reads remain local after closure; unfinished captures are not awaited. See [Migrating in v5](references/fetching/stealthy.md#migrating-in-v5) for the removed API capture interface.
## References
You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
diff --git a/agent-skill/Scrapling-Skill/examples/02_dynamic_session.py b/agent-skill/Scrapling-Skill/examples/02_browser_session.py
similarity index 79%
rename from agent-skill/Scrapling-Skill/examples/02_dynamic_session.py
rename to agent-skill/Scrapling-Skill/examples/02_browser_session.py
index 64229fe..baf6c9d 100644
--- a/agent-skill/Scrapling-Skill/examples/02_dynamic_session.py
+++ b/agent-skill/Scrapling-Skill/examples/02_browser_session.py
@@ -1,5 +1,5 @@
"""
-Example 2: Python - DynamicSession (Playwright browser automation, visible)
+Example 2: Python - StealthySession (Playwright browser automation, visible)
Scrapes all 10 pages of quotes.toscrape.com using a persistent browser session.
The browser window stays open across all page requests for efficiency.
@@ -10,11 +10,11 @@ Set headless=True to run the browser hidden.
Set disable_resources=True to skip loading images/fonts for a speed boost.
"""
-from scrapling.fetchers import DynamicSession
+from scrapling.fetchers import StealthySession
all_quotes = []
-with DynamicSession(headless=False, disable_resources=True) as session:
+with StealthySession(headless=False, disable_resources=True) as session:
for i in range(1, 11):
page = session.fetch(f"https://quotes.toscrape.com/page/{i}/")
quotes = page.css(".quote .text::text").getall()
diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md
index 4c25b22..d2dd894 100644
--- a/agent-skill/Scrapling-Skill/examples/README.md
+++ b/agent-skill/Scrapling-Skill/examples/README.md
@@ -10,7 +10,7 @@ Make sure Scrapling is installed:
```bash
pip install "scrapling[all]>=0.5"
-scrapling install --force
+scrapling install
```
## Examples
@@ -18,7 +18,7 @@ scrapling install --force
| File | Tool | Type | Best For |
|--------------------------|-------------------|-----------------------------|---------------------------------------|
| `01_fetcher_session.py` | `FetcherSession` | Python - persistent HTTP | APIs, fast multi-page scraping |
-| `02_dynamic_session.py` | `DynamicSession` | Python - browser automation | Dynamic/SPA pages |
+| `02_browser_session.py` | `StealthySession` | Python - browser automation | Dynamic/SPA pages |
| `03_stealthy_session.py` | `StealthySession` | Python - stealth browser | Cloudflare, fingerprint bypass |
| `04_spider.py` | `Spider` | Python - auto-crawling | Multi-page crawls, full-site scraping |
@@ -28,7 +28,7 @@ scrapling install --force
```bash
python examples/01_fetcher_session.py
-python examples/02_dynamic_session.py # Opens a visible browser
+python examples/02_browser_session.py # Opens a visible browser
python examples/03_stealthy_session.py # Opens a visible stealth browser
python examples/04_spider.py # Auto-crawls all pages, exports quotes.json
```
@@ -39,7 +39,6 @@ Start with the fastest, lightest option and escalate only if needed:
```
get / FetcherSession
- └─ If JS required → fetch / DynamicSession
- └─ If blocked → stealthy-fetch / StealthySession
- └─ If multi-page → Spider
+ └─ If JavaScript or anti-bot handling is needed → fetch / StealthySession
+ └─ If multi-page → Spider
```
diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
index e6e636b..a9f2739 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
@@ -1,39 +1,43 @@
# Fetchers basics
## Introduction
-Fetchers are classes that do requests or fetch pages in a single-line fashion with many features and return a [Response](#response-object) object. All fetchers have separate session classes to keep the session running (e.g., a browser fetcher keeps the browser open until you finish all requests).
+Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a [Response](#response-object) object. Starting with v0.3, all fetchers have separate classes to keep the session running, so for example, a fetcher that uses a browser will keep the browser open till you finish all your requests through it instead of opening multiple browsers. So it depends on your use case.
-Fetchers are not wrappers built on top of other libraries. They use these libraries as an engine to request/fetch pages but add features the underlying engines don't have, while still fully leveraging and optimizing them for web scraping.
+This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
+
+> Fetchers are not wrappers built on top of other libraries. However, they only use these libraries as an engine to request/fetch pages. To further clarify this, all fetchers have features that the underlying engines don't, while still fully leveraging those engines and optimizing them for Web Scraping.
## Fetchers Overview
-Scrapling provides three different fetcher classes with their session classes; each fetcher is designed for a specific use case.
+Scrapling provides HTTP fetchers (`Fetcher` and `AsyncFetcher`) and a browser fetcher (`StealthyFetcher`), with session classes for repeated requests.
The following table compares them and can be quickly used for guidance.
-| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
-|--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
-| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
-| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| JavaScript loading | ❌ | ✅ | ✅ |
-| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
-| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Small-Mid protections | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
-| Browser(s) | ❌ | Chromium and Google Chrome | Chromium and Google Chrome |
-| Browser API used | ❌ | PlayWright | PlayWright |
-| Setup Complexity | Simple | Simple | Simple |
+| Feature | Fetcher | StealthyFetcher |
+|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------------------|
+| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 |
+| Stealth | ⭐⭐ | ⭐⭐⭐⭐⭐ |
+| Anti-Bot options | ⭐⭐ | ⭐⭐⭐⭐⭐ |
+| JavaScript loading | ❌ | ✅ |
+| Memory Usage | ⭐ | ⭐⭐⭐ |
+| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
+| Browser(s) | ❌ | Chromium and Google Chrome |
+| Browser API used | ❌ | Playwright |
+| Setup Complexity | Simple | Simple |
+
+In the following pages, we will talk about each one in detail.
## Parser configuration in all fetchers
All fetchers share the same import method, as you will see in the upcoming pages
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
```
Then you use it right away without initializing like this, and it will use the default parser settings:
```python
page = StealthyFetcher.fetch('https://example.com')
```
-If you want to configure the parser ([Selector class](parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
+If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
```python
from scrapling.fetchers import Fetcher
Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest
@@ -47,9 +51,11 @@ Fetcher.keep_cdata=False # and the rest
```
Then, continue your code as usual.
-The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
+The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
-**Info:** The `adaptive` argument is disabled by default; you must enable it to use that feature.
+**Info:**
+
+The `adaptive` argument is disabled by default; you must enable it to use that feature.
### Set parser config per request
As you probably understand, the logic above for setting the parser config will apply globally to all requests/fetches made through that class, and it's intended for simplicity.
@@ -57,7 +63,7 @@ As you probably understand, the logic above for setting the parser config will a
If your use case requires a different configuration for each request/fetch, you can pass a dictionary to the request method (`fetch`/`get`/`post`/...) to an argument named `selector_config`.
## Response Object
-The `Response` object is the same as the [Selector](parsing/main_classes.md#selector) class, but it has additional details about the response, like response headers, status, cookies, etc., as shown below:
+The `Response` object is the same as the [Selector](../parsing/main_classes.md#selector) class, but it has additional details about the response, like response headers, status, cookies, etc., as shown below:
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')
@@ -72,12 +78,14 @@ page.body # Raw response body as bytes
page.encoding # Response encoding
page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
```
-All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](dynamic.md#network-history) for saved API responses and the v5 migration.
+All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](stealthy.md#network-history) for saved API responses and the v5 migration.
The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:
```python
markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)
```
-Scripts, styles, and hidden/prompt-injection content are always removed before conversion, and you can pass `css_selector` to convert specific elements only. It requires the `rag` extra (included in `ai`/`shell`/`all` too). See `../building-rag-systems.md` for the full guide.
+Scripts, styles, and hidden/prompt-injection content are always removed before conversion, and you can pass `css_selector` to convert specific elements only. It requires the `rag` extra (included in `ai`/`shell`/`all` too), and the [Building RAG systems](../building-rag-systems.md) page covers it in detail.
+
+**Note:**
-**Note:** Unlike the [Selector](parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.
\ No newline at end of file
+Unlike the [Selector](../parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.
diff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
deleted file mode 100644
index 53c1ef3..0000000
--- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
+++ /dev/null
@@ -1,474 +0,0 @@
-# Fetching dynamic websites
-
-`DynamicFetcher` (formerly `PlayWrightFetcher`) provides flexible browser automation with multiple configuration options and built-in stealth improvements.
-
-As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
-
-## Basic Usage
-You have one primary way to import this Fetcher, which is the same for all fetchers.
-
-```python
-from scrapling.fetchers import DynamicFetcher
-```
-Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
-
-**Note:** The async version of the `fetch` method is `async_fetch`.
-
-This fetcher provides three main run options that can be combined as desired.
-
-Which are:
-
-### 1. Vanilla Playwright
-```python
-DynamicFetcher.fetch('https://example.com')
-```
-Using it in that manner will open a Chromium browser and load the page. There are optimizations for speed, and some stealth goes automatically under the hood, but other than that, there are no tricks or extra features unless you enable some; it's just a plain PlayWright API.
-
-### 2. Real Chrome
-```python
-DynamicFetcher.fetch('https://example.com', real_chrome=True)
-```
-If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
-
-If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
-```commandline
-playwright install chrome
-```
-
-### 3. CDP Connection
-```python
-DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
-```
-Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
-
-The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port` (`http://localhost:9222`).
-
-
-**Notes:**
-* There was a `stealth` option here, but it was moved to the `StealthyFetcher` class, as explained on the next page, with additional features since version 0.3.13.
-* This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](stealthy.md).
-
-## Full list of arguments
-All arguments for `DynamicFetcher` and its session classes:
-
-| Argument | Description | Optional |
-|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
-| url | Target url | ❌ |
-| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
-| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
-| cookies | Set cookies for the next request. | ✔️ |
-| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
-| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
-| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
-| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
-| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
-| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
-| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
-| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |
-| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
-| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
-| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
-| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
-| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
-| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
-| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
-| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
-| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
-| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
-| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
-| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
-| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
-| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
-| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
-| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
-| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
-| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
-| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
-| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
-| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
-
-In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `blocked_domains`, `proxy`, and `selector_config`.
-
-**Notes:**
-1. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
-2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
-3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
-4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
-
-
-## Examples
-
-### Resource Control
-
-```python
-# Disable unnecessary resources
-page = DynamicFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
-```
-
-### Shadow DOM
-
-Some pages keep content inside open shadow roots, which are not included in the page's normal HTML. Enable `pierce_shadow` to include that content in the response, then use CSS or XPath selectors as usual:
-
-```python
-from scrapling.fetchers import DynamicFetcher
-
-page = DynamicFetcher.fetch('https://example.com', pierce_shadow=True)
-titles = page.css('product-card .title::text').getall()
-```
-
-The option defaults to `False`. You can also enable it for a session and disable it for an individual request:
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(pierce_shadow=True) as session:
- page = session.fetch('https://example.com')
- plain_page = session.fetch('https://example.com', pierce_shadow=False)
-```
-
-The same option works with `async_fetch`, async sessions, and stealth fetchers.
-
-Only open shadow roots are included. Closed roots and iframe documents are not traversed. Use `wait_selector` when a component loads after the page.
-
-Each open root becomes a `<shadow-root>` wrapper in the response. Slots keep their tags and attributes and contain their assigned nodes, or fallback content when nothing is assigned. Unassigned light children are omitted. The live page is unchanged.
-
-```html
-<product-card>
- <shadow-root>
- <div class="title">Product name</div>
- <slot name="price"><span class="price">20</span></slot>
- </shadow-root>
-</product-card>
-```
-
-Descendant selectors work across these elements. Direct-child selectors must include the wrappers and slots:
-
-```python
-titles = page.css('product-card .title::text').getall()
-titles = page.css('product-card > shadow-root > .title::text').getall()
-prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
-```
-
-The wrappers and slots appear in XPath paths, `.children`, `response.body`, and HTML exports. Markdown and text extraction return their content without printing the tags. Hidden slot attributes remain available to content cleaning.
-
-The snapshot follows content order, but it does not calculate CSS visibility or layout. Selectors containing `<shadow-root>` apply to the Scrapling response, not the live browser page. Other HTML parsers can handle unusual nesting differently.
-
-
-### Domain Blocking
-
-```python
-# Block requests to specific domains (and their subdomains)
-page = DynamicFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})
-```
-
-### Network Control
-
-```python
-# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
-page = DynamicFetcher.fetch('https://example.com', network_idle=True)
-
-# Custom timeout (in milliseconds)
-page = DynamicFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
-
-# Proxy support (It can also be a dictionary with only the keys 'server', 'username', and 'password'.)
-page = DynamicFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
-```
-
-### Proxy Rotation
-
-```python
-from scrapling.fetchers import DynamicSession, ProxyRotator
-
-# Set up proxy rotation
-rotator = ProxyRotator([
- "http://proxy1:8080",
- "http://proxy2:8080",
- "http://proxy3:8080",
-])
-
-# Use with session - rotates proxy automatically with each request
-with DynamicSession(proxy_rotator=rotator, headless=True) as session:
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
-
- # Override rotator for a specific request
- page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
-```
-
-**Warning:** By default, all browser-based fetchers and sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
-
-### Downloading Files
-
-```python
-page = DynamicFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
-
-with open(file='main_cover.png', mode='wb') as f:
- f.write(page.body)
-```
-
-The `body` attribute of the `Response` object always returns `bytes`.
-
-### Pre-Navigation Setup
-If you need to set up event listeners, routes, or scripts that must be registered before the page navigates, use `page_setup`. This function receives the `page` object and runs before `page.goto()` is called.
-
-```python
-from playwright.sync_api import Page
-
-def capture_websockets(page: Page):
- page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
-
-page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)
-```
-Async version:
-```python
-from playwright.async_api import Page
-
-async def capture_websockets(page: Page):
- page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
-
-page = await DynamicFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
-```
-
-You can combine it with `page_action` -- `page_setup` runs before navigation, `page_action` runs after.
-
-### Browser Automation
-This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
-
-This function is executed immediately after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, allowing it to be used for purposes beyond automation. You can alter the page as you want.
-
-In the example below, I used the pages' [mouse events](https://playwright.dev/python/docs/api/class-mouse) to scroll the page with the mouse wheel, then move the mouse.
-```python
-from playwright.sync_api import Page
-
-def scroll_page(page: Page):
- page.mouse.wheel(10, 0)
- page.mouse.move(100, 400)
- page.mouse.up()
-
-page = DynamicFetcher.fetch('https://example.com', page_action=scroll_page)
-```
-Of course, if you use the async fetch version, the function must also be async.
-```python
-from playwright.async_api import Page
-
-async def scroll_page(page: Page):
- await page.mouse.wheel(10, 0)
- await page.mouse.move(100, 400)
- await page.mouse.up()
-
-page = await DynamicFetcher.async_fetch('https://example.com', page_action=scroll_page)
-```
-
-### Wait Conditions
-
-```python
-# Wait for the selector
-page = DynamicFetcher.fetch(
- 'https://example.com',
- wait_selector='h1',
- wait_selector_state='visible'
-)
-```
-This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
-
-After that, if `load_dom` is enabled (the default), the fetcher will check again to see if all JavaScript files are loaded and executed (in the `domcontentloaded` state) or continue waiting. If you have enabled `network_idle`, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
-
-The states the fetcher can wait for can be any of the following ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
-
-- `attached`: Wait for an element to be present in the DOM.
-- `detached`: Wait for an element to not be present in the DOM.
-- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
-- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
-
-### Network History
-
-Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com', network_idle=True)
-
-for response in session.network.search(resource_type='document', limit=None):
- print(response.meta['network_id'], response.url, response.status)
- if note := response.meta.get('body_note'):
- print(note)
- else:
- print(response.body)
- print(response.css('title::text').get())
-
-session.network.clear()
-```
-
-The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-
-- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
-- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
-- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-
-Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
-
-To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
-
-The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
-
-Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-
-Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.
-
-The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.
-
-Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.
-
-By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-
-#### Migrating in v5
-
-`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
-
-### Some Stealth Features
-
-```python
-page = DynamicFetcher.fetch(
- 'https://example.com',
- google_search=True,
- useragent='Mozilla/5.0...', # Custom user agent
- locale='en-US', # Set browser locale
-)
-```
-
-### General example
-```python
-from scrapling.fetchers import DynamicFetcher
-
-def scrape_dynamic_content():
- # Use Playwright for JavaScript content
- page = DynamicFetcher.fetch(
- 'https://example.com/dynamic',
- network_idle=True,
- wait_selector='.content'
- )
-
- # Extract dynamic content
- content = page.css('.content')
-
- return {
- 'title': content.css('h1::text').get(),
- 'items': [
- item.text for item in content.css('.item')
- ]
- }
-```
-
-## Session Management
-
-To keep the browser open until you make multiple requests with the same configuration, use `DynamicSession`/`AsyncDynamicSession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-# Create a session with default configuration
-with DynamicSession(
- headless=True,
- disable_resources=True,
- real_chrome=True
-) as session:
- # Make multiple requests with the same browser instance
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
- page3 = session.fetch('https://dynamic-site.com')
-
- # All requests reuse the same tab on the same browser instance
-```
-
-### Async Session Usage
-
-```python
-import asyncio
-from scrapling.fetchers import AsyncDynamicSession
-
-async def scrape_multiple_sites():
- async with AsyncDynamicSession(
- network_idle=True,
- timeout=30000,
- max_pages=3
- ) as session:
- # Make async requests with shared browser configuration
- pages = await asyncio.gather(
- session.fetch('https://spa-app1.com'),
- session.fetch('https://spa-app2.com'),
- session.fetch('https://dynamic-content.com')
- )
- return pages
-```
-
-You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
-
-1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
-2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
-3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
-
-Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
-
-This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
-
-Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
-
-Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
-
-### Save and Load Session State
-
-Use `save_state(path)` to save cookies, local storage and IndexedDB to a JSON file. Use `load_state(path)` to restore that file in another session, or replace the current session's stored state. Both methods accept a string or `pathlib.Path` and return `None`. The format is compatible with Playwright's storage state files.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession() as session:
- session.fetch('https://example.com')
- session.save_state('state.json')
-
-with DynamicSession() as session:
- session.load_state('state.json')
- page = session.fetch('https://example.com')
-```
-
-Save after your login or other browser actions finish. Call both methods inside a started session, after any pending fetches finish. Loading replaces existing cookies, local storage and IndexedDB. Fetch or reload the page to use the restored login; loading does not navigate existing tabs.
-
-The same methods work with `StealthySession`. Use `await` with `AsyncDynamicSession` and `AsyncStealthySession`:
-
-```python
-from scrapling.fetchers import AsyncDynamicSession
-
-async def restore_session():
- async with AsyncDynamicSession() as session:
- await session.load_state('state.json')
- page = await session.fetch('https://example.com')
- await session.save_state('state.json')
- return page
-```
-
-These methods save and restore only the shared browser context. They do not affect temporary per-request proxy contexts. Sessions with `proxy_rotator` have no shared context, so these methods raise `RuntimeError`. A fixed session `proxy` works as usual.
-
-The file does not contain `sessionStorage`, open tabs or browser settings. Saving overwrites the file; its parent directory must exist. File and JSON errors are raised to the caller.
-
-### Session Benefits
-
-- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
-- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
-- **Consistent fingerprint**: Same browser fingerprint across all requests.
-- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
-
-## When to Use
-
-Use DynamicFetcher when:
-
-- Need browser automation
-- Want multiple browser options
-- Using a real Chrome browser
-- Need custom browser config
-- Want a few stealth options
-
-If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
\ No newline at end of file
diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.md
index d33fd65..516b66e 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/static.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/static.md
@@ -89,7 +89,7 @@ page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
```
-The `page` object in all cases is a [Response](choosing.md#response-object) object, which is a [Selector](parsing/main_classes.md#selector), so you can use it directly
+The `page` object in all cases is a [Response](choosing.md#response-object) object, which is a [Selector](../parsing/main_classes.md#selector), so you can use it directly
```python
>>> page.css('.something.something')
diff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
index 70e9701..63d4279 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
@@ -1,6 +1,8 @@
-# StealthyFetcher
+# Browser fetching with StealthyFetcher
-`StealthyFetcher` is a stealthy browser-based fetcher similar to [DynamicFetcher](dynamic.md), using [Playwright's API](https://playwright.dev/python/docs/intro). It adds advanced anti-bot protection bypass capabilities, most handled automatically. It shares the same browser automation model as `DynamicFetcher`, using [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) for page interaction.
+The `StealthyFetcher` class loads JavaScript pages and automates Chromium or Google Chrome with anti-bot features. It uses [Playwright's API](https://playwright.dev/python/docs/intro) with stealth patches.
+
+As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
## Basic Usage
You have one primary way to import this Fetcher, which is the same for all fetchers.
@@ -10,13 +12,48 @@ from scrapling.fetchers import StealthyFetcher
```
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
-**Note:** The async version of the `fetch` method is `async_fetch`.
+Now, we will review most of the arguments one by one, using examples. If you want to jump to a table of all arguments for quick reference, [click here](#full-list-of-arguments)
+
+**Abstract:**
+
+The async version of the `fetch` method is `async_fetch`, of course.
+
+
+This fetcher currently provides three main run options that can be combined as desired.
+
+Which are:
+
+### 1. Bundled Chromium
+```python
+StealthyFetcher.fetch('https://example.com')
+```
+This opens the bundled Chromium browser with the default stealth settings and loads the page.
+
+### 2. Real Chrome
+```python
+StealthyFetcher.fetch('https://example.com', real_chrome=True)
+```
+If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
+
+If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
+```commandline
+playwright install chrome
+```
+
+### 3. CDP Connection
+```python
+StealthyFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
+```
+Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
+
+The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser (ex: `http://localhost:9222`).
+
## What does it do?
-The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamic.md) class, and here are some of the things it does:
+StealthyFetcher supports these anti-bot features:
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
@@ -27,79 +64,275 @@ The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynami
Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
-| Argument | Description | Optional |
-|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
-| url | Target url | ❌ |
-| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
-| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
-| cookies | Set cookies for the next request. | ✔️ |
-| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
-| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
-| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
-| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
-| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
-| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
-| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
-| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |
-| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
-| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
-| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
-| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
-| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
-| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
-| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
-| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
-| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
-| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
-| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
-| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
-| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
-| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
-| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
-| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
-| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
-| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
-| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
-| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
-| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
-| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
-| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
-| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
-| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
+| Argument | Description | Optional |
+|:---------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
+| url | Target url | ❌ |
+| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
+| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
+| cookies | Set cookies for the next request. | ✔️ |
+| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
+| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
+| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
+| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
+| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
+| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
+| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
+| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |
+| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
+| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
+| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
+| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
+| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
+| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
+| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
+| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
+| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
+| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
+| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
+| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
+| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
+| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
+| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
+| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
+| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
+| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
+| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
+| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
+| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
+| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
+| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
+| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
+| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
+| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
+| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
**Notes:**
-1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
-2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
-3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
-4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
-5. `init_script` is registered with the browser context, so it runs when pages are created. Stealthy mode uses Patchright's isolated execution context by default; if your `page_action` needs to read globals that the script places on `window`, call `page.evaluate(..., isolated_context=False)` from the action.
+1. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
+2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
+3. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
+4. `init_script` is registered with the browser context, so it runs when pages are created. Stealthy mode uses Patchright's isolated execution context by default; if your `page_action` needs to read globals that the script places on `window`, call `page.evaluate(..., isolated_context=False)` from the action.
+
+## Session Management
+
+To keep the browser open until you make multiple requests with the same configuration, use `StealthySession`/`AsyncStealthySession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
+
+```python
+from scrapling.fetchers import StealthySession
+
+
+with StealthySession(
+ headless=True,
+ disable_resources=True,
+ real_chrome=True
+) as session:
+
+ page1 = session.fetch('https://example1.com')
+ page2 = session.fetch('https://example2.com')
+ page3 = session.fetch('https://dynamic-site.com')
+
+
+```
+
+### Async Session Usage
+
+```python
+import asyncio
+from scrapling.fetchers import AsyncStealthySession
+
+async def scrape_multiple_sites():
+ async with AsyncStealthySession(
+ network_idle=True,
+ timeout=30000,
+ max_pages=3
+ ) as session:
+
+ pages = await asyncio.gather(
+ session.fetch('https://spa-app1.com'),
+ session.fetch('https://spa-app2.com'),
+ session.fetch('https://dynamic-content.com')
+ )
+ return pages
+```
+
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
+
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
+
+This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
+
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
+
+### Save and Load Session State
+
+Use `save_state(path)` to save cookies, local storage and IndexedDB to a JSON file. Use `load_state(path)` to restore that file in another session, or replace the current session's stored state. Both methods accept a string or `pathlib.Path` and return `None`. The format is compatible with Playwright's storage state files.
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession() as session:
+ session.fetch('https://example.com')
+ session.save_state('state.json')
+
+with StealthySession() as session:
+ session.load_state('state.json')
+ page = session.fetch('https://example.com')
+```
+
+Save after your login or other browser actions finish. Call both methods inside a started session, after any pending fetches finish. Loading replaces existing cookies, local storage and IndexedDB. Fetch or reload the page to use the restored login; loading does not navigate existing tabs.
+
+Use `await` with `AsyncStealthySession`:
+
+```python
+from scrapling.fetchers import AsyncStealthySession
+
+async def restore_session():
+ async with AsyncStealthySession() as session:
+ await session.load_state('state.json')
+ page = await session.fetch('https://example.com')
+ await session.save_state('state.json')
+ return page
+```
+
+These methods save and restore only the shared browser context. They do not affect temporary per-request proxy contexts. Sessions with `proxy_rotator` have no shared context, so these methods raise `RuntimeError`. A fixed session `proxy` works as usual.
+
+The file does not contain `sessionStorage`, open tabs or browser settings. Saving overwrites the file; its parent directory must exist. File and JSON errors are raised to the caller.
+
+### Session Benefits
+
+- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
+- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
+- **Consistent fingerprint**: Same browser fingerprint across all requests.
+- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
+
## Examples
+It's easier to understand with examples, so let's take a look.
+
+### Resource Control
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', disable_resources=True)
+```
### Shadow DOM
-Use `pierce_shadow=True` to include open Shadow DOM content in the response HTML:
+Some pages keep content inside open shadow roots, which are not included in the page's normal HTML. Enable `pierce_shadow` to include that content in the response, then use CSS or XPath selectors as usual:
```python
+from scrapling.fetchers import StealthyFetcher
+
page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)
titles = page.css('product-card .title::text').getall()
```
-The option defaults to `False` and works with sync and async fetchers. You can set it for a session or override it for each request. Each open root adds a `<shadow-root>` wrapper to the returned HTML. Slots retain their tags and attributes; include these elements in direct-child selectors. See [Shadow DOM](dynamic.md#shadow-dom) for examples and extraction limits.
+The option defaults to `False`. You can also enable it for a session and disable it for an individual request:
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession(pierce_shadow=True) as session:
+ page = session.fetch('https://example.com')
+ plain_page = session.fetch('https://example.com', pierce_shadow=False)
+```
+The same option works with `async_fetch` and async sessions.
+
+Only open shadow roots are included. Closed roots and iframe documents are not traversed. Use `wait_selector` when a component loads after the page.
+
+Each open root becomes a `<shadow-root>` wrapper in the response. Slots keep their tags and attributes and contain their assigned nodes, or fallback content when nothing is assigned. Unassigned light children are omitted. The live page is unchanged.
+
+```html
+<product-card>
+ <shadow-root>
+ <div class="title">Product name</div>
+ <slot name="price"><span class="price">20</span></slot>
+ </shadow-root>
+</product-card>
+```
+
+Descendant selectors work across these elements. Direct-child selectors must include the wrappers and slots:
+
+```python
+titles = page.css('product-card .title::text').getall()
+titles = page.css('product-card > shadow-root > .title::text').getall()
+prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
+```
+
+The wrappers and slots appear in XPath paths, `.children`, `response.body`, and HTML exports. Markdown and text extraction return their content without printing the tags. Hidden slot attributes remain available to content cleaning.
+
+The snapshot follows content order, but it does not calculate CSS visibility or layout. Selectors containing `<shadow-root>` apply to the Scrapling response, not the live browser page. Other HTML parsers can handle unusual nesting differently.
+
+### Domain Blocking
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})
+```
+
+### Network Control
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', network_idle=True)
+
+
+page = StealthyFetcher.fetch('https://example.com', timeout=30000)
+
+
+page = StealthyFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
+```
+
+### Downloading Files
+
+```python
+page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
+
+with open(file='main_cover.png', mode='wb') as f:
+ f.write(page.body)
+```
+
+The `body` attribute of the `Response` object always returns `bytes`.
+
+### Pre-Navigation Setup
+If you need to set up event listeners, routes, or scripts that must be registered before the page navigates, use `page_setup`. This function receives the `page` object and runs before `page.goto()` is called.
+
+```python
+from patchright.sync_api import Page
+
+def capture_websockets(page: Page):
+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
+
+page = StealthyFetcher.fetch('https://example.com', page_setup=capture_websockets)
+```
+Async version:
+```python
+from patchright.async_api import Page
+
+async def capture_websockets(page: Page):
+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
+
+page = await StealthyFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
+```
+
+You can combine it with `page_action` -- `page_setup` runs before navigation, `page_action` runs after.
### Cloudflare and stealth options
```python
-# Automatic Cloudflare solver
+
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
-# Works with other stealth options
+
page = StealthyFetcher.fetch(
'https://protected-site.com',
solve_cloudflare=True,
@@ -107,7 +340,7 @@ page = StealthyFetcher.fetch(
real_chrome=True,
hide_canvas=True,
google_search=True,
- proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
+ proxy='http://username:password@host:port',
)
```
@@ -132,7 +365,7 @@ This function is executed immediately after waiting for `network_idle` (if enabl
In the example below, I used the pages' [mouse events](https://playwright.dev/python/docs/api/class-mouse) to scroll the page with the mouse wheel, then move the mouse.
```python
-from playwright.sync_api import Page
+from patchright.sync_api import Page
def scroll_page(page: Page):
page.mouse.wheel(10, 0)
@@ -143,7 +376,7 @@ page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
```
Of course, if you use the async fetch version, the function must also be async.
```python
-from playwright.async_api import Page
+from patchright.async_api import Page
async def scroll_page(page: Page):
await page.mouse.wheel(10, 0)
@@ -154,11 +387,12 @@ page = await StealthyFetcher.async_fetch('https://example.com', page_action=scro
```
### Wait Conditions
+
```python
-# Wait for the selector
+
page = StealthyFetcher.fetch(
- 'https://example.com',
- wait_selector='h1',
+ 'https://quotes.toscrape.com/js-delayed/',
+ wait_selector='.quote',
wait_selector_state='visible'
)
```
@@ -173,124 +407,111 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
+### Network History
+
+Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
-### Real-world example (Amazon)
-This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Scrapling through AI
```python
-def scrape_amazon_product(url):
- # Use StealthyFetcher to bypass protection
- page = StealthyFetcher.fetch(url)
+from scrapling.fetchers import StealthySession
- # Extract product details
- return {
- 'title': page.css('#productTitle::text').get().clean(),
- 'price': page.css('.a-price .a-offscreen::text').get(),
- 'rating': page.css('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text').get(),
- 'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
- 'features': [
- li.get().clean() for li in page.css('#feature-bullets li span::text')
- ],
- 'availability': page.css('#availability')[0].get_all_text(strip=True),
- 'images': [
- img.attrib['src'] for img in page.css('#altImages img')
- ]
- }
-```
+with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
+ session.fetch('https://example.com', network_idle=True)
-## Session Management
+for response in session.network.search(resource_type='document', limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ if note := response.meta.get('body_note'):
+ print(note)
+ else:
+ print(response.body)
+ print(response.css('title::text').get())
-To keep the browser open until you make multiple requests with the same configuration, use `StealthySession`/`AsyncStealthySession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
+session.network.clear()
+```
-```python
-from scrapling.fetchers import StealthySession
+The same API works with `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-# Create a session with default configuration
-with StealthySession(
- headless=True,
- real_chrome=True,
- block_webrtc=True,
- solve_cloudflare=True
-) as session:
- # Make multiple requests with the same browser instance
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
- page3 = session.fetch('https://nopecha.com/demo/cloudflare')
-
- # All requests reuse the same tab on the same browser instance
-```
+- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
+- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
+- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-### Async Session Usage
+Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
-```python
-import asyncio
-from scrapling.fetchers import AsyncStealthySession
+To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
-async def scrape_multiple_sites():
- async with AsyncStealthySession(
- real_chrome=True,
- block_webrtc=True,
- solve_cloudflare=True,
- timeout=60000, # 60 seconds for Cloudflare challenges
- max_pages=3
- ) as session:
- # Make async requests with shared browser configuration
- pages = await asyncio.gather(
- session.fetch('https://site1.com'),
- session.fetch('https://site2.com'),
- session.fetch('https://protected-site.com')
- )
- return pages
-```
+The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
-You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
+Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
-2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
-3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.
-Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
+The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.
-This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
+Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.
-Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
+#### Migrating in v5
-### Network History
+`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
-Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to keep completed requests across page loads, actions, tabs, and temporary proxy contexts. Find API calls, HTTP errors, and redirects, then read saved Scrapling responses without sending requests again. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included. Saved headers and bodies remain readable after the session closes.
+### Browser Headers and Locale
```python
-from scrapling.fetchers import StealthySession
+page = StealthyFetcher.fetch(
+ 'https://example.com',
+ google_search=True,
+ useragent='Mozilla/5.0...',
+ locale='en-US',
+)
+```
-with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com', network_idle=True)
+### General example
+```python
+from scrapling.fetchers import StealthyFetcher
-for response in session.network.search(resource_type='document', limit=None):
- if note := response.meta.get('body_note'):
- print(note)
- else:
- print(response.body)
+def scrape_dynamic_content():
+
+ page = StealthyFetcher.fetch(
+ 'https://example.com/dynamic',
+ network_idle=True,
+ wait_selector='.content'
+ )
+
+
+ content = page.css('.content')
+
+ return {
+ 'title': content.css('h1::text').get(),
+ 'items': [
+ item.text for item in content.css('.item')
+ ]
+ }
```
-Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. `search()` and `get()` return saved `Response` objects directly. Their `.meta` contains `network_id`, `resource_type`, and `request_body`; read `.body`, `.headers`, `.request_headers`, `.status`, and `.url` on the response itself. Search returns at most 100 matches by default; `limit=None` returns all retained matches. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
+### Proxy Rotation
-The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
+```python
+from scrapling.fetchers import StealthySession, ProxyRotator
-Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. See [Migrating in v5](dynamic.md#migrating-in-v5) for the API capture changes.
+rotator = ProxyRotator([
+ "http://proxy1:8080",
+ "http://proxy2:8080",
+ "http://proxy3:8080",
+])
-### Save and Load Session State
-`StealthySession` supports `session.save_state('state.json')` and `session.load_state('state.json')`. Use `await` for both methods with `AsyncStealthySession`. The JSON file contains cookies, local storage and IndexedDB; loading replaces the current stored state. Call these methods inside a started session, after pending fetches finish. `sessionStorage` is not saved, and sessions with `proxy_rotator` are not supported. See [Save and Load Session State](dynamic.md#save-and-load-session-state) for examples and scope.
+with StealthySession(proxy_rotator=rotator, headless=True) as session:
+ page1 = session.fetch('https://example1.com')
+ page2 = session.fetch('https://example2.com')
-### Session Benefits
-- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
-- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
-- **Consistent fingerprint**: Same browser fingerprint across all requests.
-- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
+ page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
+```
+
+**Warning:**
+
+Remember that by default, all browser-based fetchers and sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
## When to Use
@@ -301,4 +522,4 @@ Use StealthyFetcher when:
- Full JavaScript support needed
- Want automatic stealth features
- Need browser automation
-- Dealing with Cloudflare protection
\ No newline at end of file
+- Dealing with Cloudflare protection
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 5f5d898..66de6fc 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -6,7 +6,7 @@ Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int
## Shadow DOM
-Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](fetching/dynamic.md#shadow-dom) for selector examples and limits.
+Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](fetching/stealthy.md#shadow-dom) for selector examples and limits.
## One-shot tools
@@ -465,7 +465,7 @@ In a Claude Desktop configuration, add the option to the server arguments:
}
```
-You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Pass `executable_path` to `browser_fetch_once` or `browser_open` when a single fetch or session needs a different browser executable. The `scrapling extract fetch` and `scrapling extract stealthy-fetch` CLI commands support the same `--executable-path` option and environment variable fallback.
+You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Pass `executable_path` to `browser_fetch_once` or `browser_open` when a single fetch or session needs a different browser executable. The `scrapling extract fetch` CLI command supports the same `--executable-path` option and environment variable fallback.
The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling-mcp` binary with no arguments (or the `scrapling` binary with `mcp` as the argument on versions before 0.4.13).
diff --git a/agent-skill/Scrapling-Skill/references/spiders/advanced.md b/agent-skill/Scrapling-Skill/references/spiders/advanced.md
index 0d21e72..bdb2333 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/advanced.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/advanced.md
@@ -211,7 +211,7 @@ Key differences from `start()`:
- Items are yielded one by one as they're scraped, not collected into a list
- You can access `spider.stats` during iteration for real-time statistics
-**Note:** The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results--statistics).
+**Note:** The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results-statistics).
You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
index 68ea7b8..11fd8ef 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
@@ -51,7 +51,6 @@ A priority queue with built-in URL deduplication. Requests are fingerprinted bas
Manages one or more named session instances. Each session is one of:
- [FetcherSession](../fetching/static.md)
-- [AsyncDynamicSession](../fetching/dynamic.md)
- [AsyncStealthySession](../fetching/stealthy.md)
When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started with the spider start (default) or lazily (started on the first use).
diff --git a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
index d9aa7d4..536b7f0 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
@@ -34,7 +34,7 @@ Each request automatically gets the next proxy in the rotation. The proxy used i
Browser sessions support both string and dict proxy formats:
```python
-from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, ProxyRotator
+from scrapling.fetchers import AsyncStealthySession, ProxyRotator
# String proxies work for all session types
rotator = ProxyRotator([
diff --git a/agent-skill/Scrapling-Skill/references/spiders/sessions.md b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
index 8ab54ea..e9b793a 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/sessions.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
@@ -12,8 +12,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md
| Session Type | Use Case |
|-------------------------------------------------|------------------------------------------|
| [FetcherSession](../fetching/static.md) | Fast HTTP requests, no JavaScript |
-| [AsyncDynamicSession](../fetching/dynamic.md) | Browser automation, JavaScript rendering |
-| [AsyncStealthySession](../fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc. |
+| [AsyncStealthySession](../fetching/stealthy.md) | JavaScript, browser automation, anti-bot bypass |
## Configuring Sessions
@@ -158,7 +157,7 @@ async def parse(self, response: Response):
**Warning:** When using `FetcherSession` in spiders, you cannot use `.get()` and `.post()` methods directly. By default, the request is an HTTP GET request; to use another HTTP method, pass it to the `method` argument as in the above example. This unifies the `Request` interface across all session types.
-For browser sessions (`AsyncDynamicSession`, `AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
+For browser sessions (`AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
```python
async def parse(self, response: Response):
diff --git a/docs/README_AR.md b/docs/README_AR.md
index bd78404..468ca89 100644
--- a/docs/README_AR.md
+++ b/docs/README_AR.md
@@ -56,7 +56,7 @@ Scrapling هو إطار عمل تكيفي لـ Web Scraping يتعامل مع ك
زحف سريع للغاية مع إحصائيات فورية و Streaming. مبني بواسطة مستخرجي الويب لمستخرجي الويب والمستخدمين العاديين، هناك شيء للجميع.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # احصل على الموقع بشكل خفي!
products = p.css('.product', auto_save=True) # استخرج بيانات تنجو من تغييرات تصميم الموقع!
@@ -243,9 +243,9 @@ MySpider().start()
### جلب متقدم للمواقع مع دعم الجلسات
- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP/3.
-- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `DynamicFetcher` التي تدعم Chromium من Playwright و Google Chrome.
+- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `StealthyFetcher` التي تدعم Chromium من Playwright و Google Chrome.
- **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` وانتحال fingerprint. يمكنه تجاوز جميع أنواع Turnstile/Interstitial من Cloudflare بسهولة بالأتمتة.
-- **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` و`DynamicSession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات.
+- **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات.
- **تدوير Proxy**: `ProxyRotator` مدمج مع استراتيجيات التدوير الدوري أو المخصصة عبر جميع أنواع الجلسات، بالإضافة إلى تجاوزات Proxy لكل طلب.
- **حظر النطاقات والإعلانات**: حظر الطلبات إلى نطاقات محددة (ونطاقاتها الفرعية) أو تفعيل حظر الإعلانات المدمج (~3,500 نطاق إعلانات/تتبع معروف) في الجوالب المعتمدة على المتصفح.
- **منع تسرب DNS**: دعم اختياري لـ DNS-over-HTTPS لتوجيه استعلامات DNS عبر Cloudflare DoH، مما يمنع تسرب DNS عند استخدام Proxy.
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **البحث عن عناصر مشابهة**: تحديد العناصر المشابهة للعناصر الموجودة تلقائياً.
### ميزات الذكاء الاصطناعي
-- 🤖 **خادم MCP**: دع روبوتات الدردشة ووكلاء الذكاء الاصطناعي (Claude/Cursor/إلخ) يستخرجون البيانات عبر Scrapling بأدوات تعمل لمرة واحدة أو عبر جلسات، وتغطي طلبات HTTP العادية (بأي طريقة)، وجلب الصفحات بالمتصفح، والجلب الخفي الذي يتجاوز Cloudflare. يتم تضييق الصفحات بمحددات CSS وتنظيفها من محتوى حقن الأوامر قبل أن يراها الذكاء الاصطناعي، فيقرأ الوكيل أقل، ويكلف أقل، ولا يمكن اختطافه بنص مخفي. كما تتوفر لقطات الشاشة، والمتصفحات البعيدة عبر CDP، ونقل HTTP آمن افتراضياً. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **خادم MCP**: دع روبوتات الدردشة ووكلاء الذكاء الاصطناعي (Claude/Cursor/إلخ) يستخرجون البيانات عبر Scrapling بأدوات تعمل لمرة واحدة أو عبر جلسات، وتغطي طلبات HTTP العادية (بأي طريقة) وجلب الصفحات بالمتصفح الخفي الذي يتجاوز Cloudflare. يتم تضييق الصفحات بمحددات CSS وتنظيفها من محتوى حقن الأوامر قبل أن يراها الذكاء الاصطناعي، فيقرأ الوكيل أقل، ويكلف أقل، ولا يمكن اختطافه بنص مخفي. كما تتوفر لقطات الشاشة، والمتصفحات البعيدة عبر CDP، ونقل HTTP آمن افتراضياً. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: [مهارة جاهزة للتثبيت](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) تُعلّم وكلاء البرمجة المكتبة بالكامل، حتى يكون الكود الذي يكتبونه بـ Scrapling مطابقاً للواجهة الحالية بدلاً من التخمين.
- 📚 **Markdown جاهز لأنظمة RAG**: حوّل أي صفحة إلى Markdown نظيف ومعقّم وجاهز لنماذج اللغة بسطر واحد (`page.markdown()`)، أو ازحف موقعاً كاملاً إلى مجموعة ملفات Markdown باستخدام القالب `SiteToMarkdownSpider`، كل ذلك بدون استخدام أي نموذج لغوي في العملية. ([الوثائق](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
أتمتة المتصفح الكاملة
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # أبقِ المتصفح مفتوحاً حتى تنتهي
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # أبقِ المتصفح مفتوحاً حتى تنتهي
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # محدد XPath إذا كنت تفضله
# أو استخدم نمط الطلب لمرة واحدة، يفتح المتصفح لهذا الطلب، ثم يغلقه بعد الانتهاء
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ page = Selector("<html>...</html>")
### أمثلة إدارة الجلسات بشكل Async
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` واعٍ بالسياق ويعمل في كلا النمطين المتزامن/async
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ scrapling shell
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # جميع العناصر المطابقة لمحدد CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_CN.md b/docs/README_CN.md
index d2f7447..0b318ff 100644
--- a/docs/README_CN.md
+++ b/docs/README_CN.md
@@ -56,7 +56,7 @@ Scrapling 是一个自适应 Web Scraping 框架,能处理从单个请求到
极速爬取,实时统计和 Streaming。由 Web Scraper 为 Web Scraper 和普通用户而构建,每个人都能找到适合自己的功能。
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # 隐秘地获取网站!
products = p.css('.product', auto_save=True) # 抓取在网站设计变更后仍能存活的数据!
@@ -243,9 +243,9 @@ MySpider().start()
### 支持 Session 的高级网站获取
- **HTTP 请求**:使用 `Fetcher` 类进行快速和隐秘的 HTTP 请求。可以模拟浏览器的 TLS fingerprint、标头并使用 HTTP/3。
-- **动态加载**:通过 `DynamicFetcher` 类使用完整的浏览器自动化获取动态网站,支持 Playwright 的 Chromium 和 Google Chrome。
+- **动态加载**:通过 `StealthyFetcher` 类使用完整的浏览器自动化获取动态网站,支持 Playwright 的 Chromium 和 Google Chrome。
- **反机器人绕过**:使用 `StealthyFetcher` 的高级隐秘功能和 fingerprint 伪装。可以轻松自动绕过所有类型的 Cloudflare Turnstile/Interstitial。
-- **Session 管理**:使用 `FetcherSession`、`StealthySession` 和 `DynamicSession` 类实现持久化 Session 支持,用于跨请求的 cookie 和状态管理。
+- **Session 管理**:使用 `FetcherSession` 和 `StealthySession` 类实现持久化 Session 支持,用于跨请求的 cookie 和状态管理。
- **Proxy 轮换**:内置 `ProxyRotator`,支持轮询或自定义策略,适用于所有 Session 类型,并支持按请求覆盖 Proxy。
- **域名和广告屏蔽**:在基于浏览器的 Fetcher 中屏蔽对特定域名(及其子域名)的请求,或启用内置广告屏蔽(约 3,500 个已知广告/追踪域名)。
- **DNS 泄漏防护**:可选的 DNS-over-HTTPS 支持,通过 Cloudflare 的 DoH 路由 DNS 查询,防止使用代理时的 DNS 泄漏。
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **查找相似元素**:自动定位与已找到元素相似的元素。
### AI 功能
-- 🤖 **MCP 服务器**:让 AI 聊天机器人和智能体(Claude/Cursor 等)通过 Scrapling 抓取网页,提供一次性工具和基于会话的工具,覆盖普通 HTTP 请求(任意方法)、浏览器抓取以及可绕过 Cloudflare 的隐身抓取。页面在交给 AI 之前会先用 CSS 选择器缩小范围并清除提示注入内容,因此智能体读得更少、花得更少,也不会被隐藏文本劫持。还包含页面截图、通过 CDP 控制远程浏览器,以及默认安全的 HTTP 传输。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP 服务器**:让 AI 聊天机器人和智能体(Claude/Cursor 等)通过 Scrapling 抓取网页,提供一次性工具和基于会话的工具,覆盖普通 HTTP 请求(任意方法)以及可绕过 Cloudflare 的隐身浏览器抓取。页面在交给 AI 之前会先用 CSS 选择器缩小范围并清除提示注入内容,因此智能体读得更少、花得更少,也不会被隐藏文本劫持。还包含页面截图、通过 CDP 控制远程浏览器,以及默认安全的 HTTP 传输。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:[开箱即用的 Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html),让编码智能体全面掌握本库,使它们用 Scrapling 写出的代码符合当前 API,而不是靠猜测。
- 📚 **RAG 就绪的 Markdown**:用一行代码(`page.markdown()`)把任意页面转换为干净、经过净化、可直接供 LLM 使用的 Markdown,或使用 `SiteToMarkdownSpider` 模板把整个网站抓取为 Markdown 语料库,整个过程无需调用 LLM。([文档](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
完整的浏览器自动化
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # 保持浏览器打开直到完成
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # 保持浏览器打开直到完成
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # 如果您偏好 XPath 选择器
# 或使用一次性请求样式,为此请求打开浏览器,完成后关闭
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ page = Selector("<html>...</html>")
### Async Session 管理示例
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession`是上下文感知的,可以在 sync/async 模式下工作
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ scrapling shell
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # 所有匹配 CSS 选择器'#fromSkipToProducts' 的元素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_DE.md b/docs/README_DE.md
index 5a56203..4d89f5e 100644
--- a/docs/README_DE.md
+++ b/docs/README_DE.md
@@ -56,7 +56,7 @@ Sein Parser lernt aus Website-Änderungen und lokalisiert Ihre Elemente automati
Blitzschnelle Crawls mit Echtzeit-Statistiken und Streaming. Von Web Scrapern für Web Scraper und normale Benutzer entwickelt, ist für jeden etwas dabei.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Website unbemerkt abrufen!
products = p.css('.product', auto_save=True) # Daten scrapen, die Website-Designänderungen überleben!
@@ -243,9 +243,9 @@ MySpider().start()
### Erweitertes Website-Abrufen mit Session-Unterstützung
- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerprints und Header imitieren und HTTP/3 verwenden.
-- **Dynamisches Laden**: Dynamische Websites mit vollständiger Browser-Automatisierung über die `DynamicFetcher`-Klasse abrufen, die Playwrights Chromium und Google Chrome unterstützt.
+- **Dynamisches Laden**: Dynamische Websites mit vollständiger Browser-Automatisierung über die `StealthyFetcher`-Klasse abrufen, die Playwrights Chromium und Google Chrome unterstützt.
- **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` und Fingerprint-Spoofing. Kann alle Arten von Cloudflares Turnstile/Interstitial einfach mit Automatisierung umgehen.
-- **Session-Verwaltung**: Persistente Session-Unterstützung mit den Klassen `FetcherSession`, `StealthySession` und `DynamicSession` für Cookie- und Zustandsverwaltung über Anfragen hinweg.
+- **Session-Verwaltung**: Persistente Session-Unterstützung mit den Klassen `FetcherSession` und `StealthySession` für Cookie- und Zustandsverwaltung über Anfragen hinweg.
- **Proxy-Rotation**: Integrierter `ProxyRotator` mit zyklischen oder benutzerdefinierten Rotationsstrategien über alle Session-Typen hinweg, plus Proxy-Überschreibungen pro Anfrage.
- **Domain- & Werbeblockierung**: Anfragen an bestimmte Domains (und deren Subdomains) blockieren oder die integrierte Werbeblockierung (~3.500 bekannte Werbe-/Tracker-Domains) in browserbasierten Fetchern aktivieren.
- **DNS-Leak-Prävention**: Optionale DNS-over-HTTPS-Unterstützung zur Weiterleitung von DNS-Anfragen über Cloudflares DoH, um DNS-Leaks bei der Verwendung von Proxys zu verhindern.
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **Ähnliche Elemente finden**: Elemente, die gefundenen Elementen ähnlich sind, automatisch lokalisieren.
### KI-Funktionen
-- 🤖 **MCP-Server**: Lassen Sie KI-Chatbots und -Agenten (Claude/Cursor/etc.) über Scrapling scrapen, mit Einmal- oder sitzungsbasierten Tools für einfache HTTP-Anfragen (jede Methode), Browser-Abrufe und Stealth-Abrufe, die Cloudflare umgehen. Seiten werden mit CSS-Selektoren eingegrenzt und von Prompt-Injection-Inhalten bereinigt, bevor die KI sie sieht, sodass der Agent weniger liest, weniger kostet und nicht durch versteckten Text gekapert werden kann. Screenshots, Remote-Browser über CDP und ein standardmäßig abgesicherter HTTP-Transport sind enthalten. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP-Server**: Lassen Sie KI-Chatbots und -Agenten (Claude/Cursor/etc.) über Scrapling scrapen, mit Einmal- oder sitzungsbasierten Tools für einfache HTTP-Anfragen (jede Methode) und Stealth-Browser-Abrufe, die Cloudflare umgehen. Seiten werden mit CSS-Selektoren eingegrenzt und von Prompt-Injection-Inhalten bereinigt, bevor die KI sie sieht, sodass der Agent weniger liest, weniger kostet und nicht durch versteckten Text gekapert werden kann. Screenshots, Remote-Browser über CDP und ein standardmäßig abgesicherter HTTP-Transport sind enthalten. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Ein installationsfertiger [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), der Coding-Agents die gesamte Bibliothek beibringt, damit der Code, den sie mit Scrapling schreiben, zur aktuellen API passt, statt geraten zu werden.
- 📚 **RAG-fertiges Markdown**: Wandeln Sie jede Seite mit einer Zeile (`page.markdown()`) in sauberes, bereinigtes, LLM-fertiges Markdown um oder crawlen Sie eine ganze Website mit der Vorlage `SiteToMarkdownSpider` in einen Markdown-Korpus, ganz ohne LLM im Prozess. ([Dokumentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
Vollständige Browser-Automatisierung
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Browser offen halten, bis Sie fertig sind
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Browser offen halten, bis Sie fertig sind
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # XPath-Selektor, falls bevorzugt
# Oder einmaligen Anfragenstil verwenden: öffnet den Browser für diese Anfrage und schließt ihn nach Abschluss
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ Und es funktioniert genau auf die gleiche Weise!
### Beispiele für async Session-Verwaltung
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` ist kontextbewusst und kann sowohl in sync- als auch in async-Mustern arbeiten
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ Seiten direkt ohne Programmierung in eine Datei extrahieren (extrahiert standard
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Alle Elemente, die dem CSS-Selektor '#fromSkipToProducts' entsprechen
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_ES.md b/docs/README_ES.md
index 3ce5b02..1efb925 100644
--- a/docs/README_ES.md
+++ b/docs/README_ES.md
@@ -56,7 +56,7 @@ Su parser aprende de los cambios de los sitios web y relocaliza automáticamente
Rastreos ultrarrápidos con estadísticas en tiempo real y Streaming. Construido por Web Scrapers para Web Scrapers y usuarios regulares, hay algo para todos.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # ¡Obtén el sitio web bajo el radar!
products = p.css('.product', auto_save=True) # ¡Extrae datos que sobreviven a cambios de diseño del sitio web!
@@ -243,9 +243,9 @@ MySpider().start()
### Obtención Avanzada de Sitios Web con Soporte de Session
- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar el fingerprint TLS de los navegadores, encabezados y usar HTTP/3.
-- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `DynamicFetcher` compatible con Chromium de Playwright y Google Chrome.
+- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `StealthyFetcher` compatible con Chromium de Playwright y Google Chrome.
- **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` y falsificación de fingerprint. Puede evadir fácilmente todos los tipos de Turnstile/Interstitial de Cloudflare con automatización.
-- **Gestión de Session**: Soporte de sesión persistente con las clases `FetcherSession`, `StealthySession` y `DynamicSession` para la gestión de cookies y estado entre solicitudes.
+- **Gestión de Session**: Soporte de sesión persistente con las clases `FetcherSession` y `StealthySession` para la gestión de cookies y estado entre solicitudes.
- **Rotación de Proxy**: `ProxyRotator` integrado con estrategias de rotación cíclica o personalizadas en todos los tipos de sesión, además de sobrescrituras de Proxy por solicitud.
- **Bloqueo de Dominios y Anuncios**: Bloquea solicitudes a dominios específicos (y sus subdominios) o activa el bloqueo de anuncios integrado (~3,500 dominios de anuncios/rastreadores conocidos) en fetchers basados en navegador.
- **Prevención de Fugas DNS**: Soporte opcional de DNS-over-HTTPS para enrutar consultas DNS a través del DoH de Cloudflare, previniendo fugas DNS al usar proxies.
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **Encontrar Elementos Similares**: Localiza automáticamente elementos similares a los elementos encontrados.
### Características de IA
-- 🤖 **Servidor MCP**: Permite que los chatbots y agentes de IA (Claude/Cursor/etc) hagan scraping a través de Scrapling con herramientas de un solo uso o basadas en sesiones que cubren peticiones HTTP simples (cualquier método), cargas con navegador y cargas sigilosas que evitan Cloudflare. Las páginas se acotan con selectores CSS y se limpian de contenido de inyección de prompts antes de que la IA las vea, así que el agente lee menos, cuesta menos y no puede ser secuestrado por texto oculto. Incluye capturas de pantalla, navegadores remotos por CDP y un transporte HTTP seguro por defecto. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Servidor MCP**: Permite que los chatbots y agentes de IA (Claude/Cursor/etc) hagan scraping a través de Scrapling con herramientas de un solo uso o basadas en sesiones que cubren peticiones HTTP simples (cualquier método) y cargas con navegador sigiloso que evitan Cloudflare. Las páginas se acotan con selectores CSS y se limpian de contenido de inyección de prompts antes de que la IA las vea, así que el agente lee menos, cuesta menos y no puede ser secuestrado por texto oculto. Incluye capturas de pantalla, navegadores remotos por CDP y un transporte HTTP seguro por defecto. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) listo para instalar que enseña la biblioteca completa a los agentes de programación, para que el código que escriban con Scrapling coincida con la API actual en vez de adivinarla.
- 📚 **Markdown listo para RAG**: Convierte cualquier página en Markdown limpio, saneado y listo para LLM con una sola línea (`page.markdown()`), o rastrea un sitio web completo hacia un corpus de Markdown con la plantilla `SiteToMarkdownSpider`, todo sin un LLM en el proceso. ([documentación](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
Automatización completa del navegador
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Mantén el navegador abierto hasta que termines
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Mantén el navegador abierto hasta que termines
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # Selector XPath si lo prefieres
# O usa el estilo de solicitud de una sola vez, abre el navegador para esta solicitud, luego lo cierra después de terminar
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ page = Selector("<html>...</html>")
### Ejemplos de Gestión de Session Async
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` es consciente del contexto y puede funcionar tanto en patrones sync/async
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ Extraer páginas a un archivo directamente sin programar (Extrae el contenido de
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos los elementos que coinciden con el selector CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_FR.md b/docs/README_FR.md
index 95fe8d1..6c786ac 100644
--- a/docs/README_FR.md
+++ b/docs/README_FR.md
@@ -56,7 +56,7 @@ Son parser apprend des modifications de sites web et relocalise automatiquement
Des crawls ultra-rapides avec des statistiques en temps réel et du streaming. Conçu par des Web Scrapers pour des Web Scrapers et des utilisateurs réguliers, il y en a pour tout le monde.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Récupérer un site web en toute discrétion !
products = p.css('.product', auto_save=True) # Scraper des données qui survivent aux changements de design !
@@ -243,9 +243,9 @@ MySpider().start()
### Récupération avancée de sites web avec support de sessions
- **Requêtes HTTP** : Requêtes HTTP rapides et furtives avec la classe `Fetcher`. Peut imiter l'empreinte TLS des navigateurs, les headers et utiliser HTTP/3.
-- **Chargement dynamique** : Récupérez des sites web dynamiques avec une automatisation complète du navigateur via la classe `DynamicFetcher` supportant Chromium de Playwright et Google Chrome.
+- **Chargement dynamique** : Récupérez des sites web dynamiques avec une automatisation complète du navigateur via la classe `StealthyFetcher` supportant Chromium de Playwright et Google Chrome.
- **Contournement anti-bot** : Capacités de furtivité avancées avec `StealthyFetcher` et usurpation d'empreinte. Peut facilement contourner tous les types de Turnstile/Interstitial de Cloudflare avec l'automatisation.
-- **Gestion de sessions** : Support de sessions persistantes avec les classes `FetcherSession`, `StealthySession` et `DynamicSession` pour la gestion des cookies et de l'état entre les requêtes.
+- **Gestion de sessions** : Support de sessions persistantes avec les classes `FetcherSession` et `StealthySession` pour la gestion des cookies et de l'état entre les requêtes.
- **Rotation de proxy** : `ProxyRotator` intégré avec des stratégies de rotation cycliques ou personnalisées sur tous les types de sessions, plus des surcharges de proxy par requête.
- **Blocage de domaines et publicités** : Bloquez les requêtes vers des domaines spécifiques (et leurs sous-domaines) ou activez le blocage de publicités intégré (~3 500 domaines publicitaires/traceurs connus) dans les fetchers basés sur navigateur.
- **Prévention des fuites DNS** : Support optionnel de DNS-over-HTTPS pour router les requêtes DNS via le DoH de Cloudflare, empêchant les fuites DNS lors de l'utilisation de proxies.
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **Trouver des éléments similaires** : Localisez automatiquement des éléments similaires aux éléments trouvés.
### Fonctionnalités IA
-- 🤖 **Serveur MCP** : Laissez les chatbots et agents IA (Claude/Cursor/etc.) scraper via Scrapling avec des outils ponctuels ou basés sur des sessions couvrant les requêtes HTTP simples (toute méthode), les chargements par navigateur et les chargements furtifs qui contournent Cloudflare. Les pages sont réduites avec des sélecteurs CSS et nettoyées du contenu d'injection de prompt avant que l'IA ne les voie, donc l'agent lit moins, coûte moins et ne peut pas être détourné par du texte caché. Captures d'écran, navigateurs distants via CDP et transport HTTP sécurisé par défaut sont inclus. ([vidéo de démonstration](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Serveur MCP** : Laissez les chatbots et agents IA (Claude/Cursor/etc.) scraper via Scrapling avec des outils ponctuels ou basés sur des sessions couvrant les requêtes HTTP simples (toute méthode) et les chargements par navigateur furtif qui contournent Cloudflare. Les pages sont réduites avec des sélecteurs CSS et nettoyées du contenu d'injection de prompt avant que l'IA ne les voie, donc l'agent lit moins, coûte moins et ne peut pas être détourné par du texte caché. Captures d'écran, navigateurs distants via CDP et transport HTTP sécurisé par défaut sont inclus. ([vidéo de démonstration](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill** : Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) prêt à installer qui enseigne toute la bibliothèque aux agents de codage, pour que le code qu'ils écrivent avec Scrapling corresponde à l'API actuelle au lieu d'être deviné.
- 📚 **Markdown prêt pour le RAG** : Transformez n'importe quelle page en Markdown propre, assaini et prêt pour les LLM en une ligne (`page.markdown()`), ou crawlez un site entier en un corpus Markdown avec le template `SiteToMarkdownSpider`, le tout sans LLM dans la boucle. ([documentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
Automatisation complète du navigateur
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Garder le navigateur ouvert jusqu'à ce que vous ayez terminé
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Garder le navigateur ouvert jusqu'à ce que vous ayez terminé
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # Sélecteur XPath si vous le préférez
# Ou utiliser le style requête ponctuelle : ouvre le navigateur pour cette requête, puis le ferme après
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ Et cela fonctionne exactement de la même manière !
### Exemples de gestion de sessions async
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` est sensible au contexte et peut fonctionner en mode sync comme async
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ Extraire des pages directement dans un fichier sans programmation (extrait par d
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Tous les éléments correspondant au sélecteur CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_JP.md b/docs/README_JP.md
index a5df603..fc640ed 100644
--- a/docs/README_JP.md
+++ b/docs/README_JP.md
@@ -56,7 +56,7 @@ Scrapling は、単一のリクエストから本格的なクロールまです
リアルタイム統計と Streaming による超高速クロール。Web Scraper によって、Web Scraper と一般ユーザーのために構築され、誰にでも何かがあります。
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # レーダーの下でウェブサイトを取得!
products = p.css('.product', auto_save=True) # ウェブサイトのデザイン変更に耐えるデータをスクレイプ!
@@ -243,9 +243,9 @@ MySpider().start()
### Session サポート付き高度なウェブサイト取得
- **HTTP リクエスト**:`Fetcher` クラスで高速かつステルスな HTTP リクエスト。ブラウザの TLS fingerprint、ヘッダーを模倣し、HTTP/3 を使用可能。
-- **動的読み込み**:Playwright の Chromium と Google Chrome をサポートする `DynamicFetcher` クラスによる完全なブラウザ自動化で動的ウェブサイトを取得。
+- **動的読み込み**:Playwright の Chromium と Google Chrome をサポートする `StealthyFetcher` クラスによる完全なブラウザ自動化で動的ウェブサイトを取得。
- **アンチボット回避**:`StealthyFetcher` と fingerprint 偽装による高度なステルス機能。自動化で Cloudflare の Turnstile/Interstitial のすべてのタイプを簡単に回避。
-- **Session 管理**:リクエスト間で Cookie と状態を管理するための `FetcherSession`、`StealthySession`、`DynamicSession` クラスによる永続的な Session サポート。
+- **Session 管理**:リクエスト間で Cookie と状態を管理するための `FetcherSession`、`StealthySession` クラスによる永続的な Session サポート。
- **Proxy 回転**:すべての Session タイプに対応したラウンドロビンまたはカスタム戦略の組み込み `ProxyRotator`、さらにリクエストごとの Proxy オーバーライド。
- **ドメイン&広告ブロック**:ブラウザベースの Fetcher で特定のドメイン(およびそのサブドメイン)へのリクエストをブロック、または内蔵広告ブロック(約3,500の既知の広告/トラッカードメイン)を有効化。
- **DNS リーク防止**:Proxy 使用時の DNS リークを防ぐため、Cloudflare の DoH 経由で DNS クエリをルーティングするオプションの DNS-over-HTTPS サポート。
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **類似要素の検出**:見つかった要素に類似した要素を自動的に特定。
### AI 機能
-- 🤖 **MCP サーバー**:AI チャットボットやエージェント(Claude/Cursor など)が Scrapling を通じてスクレイピングできるようにします。単発ツールとセッションベースのツールがあり、通常の HTTP リクエスト(任意のメソッド)、ブラウザ取得、Cloudflare を回避するステルス取得をカバーします。ページは AI に渡す前に CSS セレクターで絞り込まれ、プロンプトインジェクションの内容が除去されるため、エージェントは読む量もコストも減り、隠しテキストに乗っ取られることもありません。スクリーンショット、CDP 経由のリモートブラウザ、デフォルトで安全な HTTP トランスポートも含まれます。([デモ動画](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP サーバー**:AI チャットボットやエージェント(Claude/Cursor など)が Scrapling を通じてスクレイピングできるようにします。単発ツールとセッションベースのツールがあり、通常の HTTP リクエスト(任意のメソッド)と、Cloudflare を回避するステルスブラウザ取得をカバーします。ページは AI に渡す前に CSS セレクターで絞り込まれ、プロンプトインジェクションの内容が除去されるため、エージェントは読む量もコストも減り、隠しテキストに乗っ取られることもありません。スクリーンショット、CDP 経由のリモートブラウザ、デフォルトで安全な HTTP トランスポートも含まれます。([デモ動画](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:ライブラリ全体をコーディングエージェントに教える、インストールするだけの [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)。エージェントが Scrapling で書くコードが、推測ではなく現在の API に沿ったものになります。
- 📚 **RAG 対応の Markdown**:1 行(`page.markdown()`)で任意のページをクリーンでサニタイズ済みの LLM 向け Markdown に変換したり、`SiteToMarkdownSpider` テンプレートでサイト全体を Markdown コーパスとしてクロールしたりできます。処理に LLM は一切使いません。([ドキュメント](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
完全なブラウザ自動化
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # 完了するまでブラウザを開いたままにする
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # 完了するまでブラウザを開いたままにする
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # お好みであれば XPath セレクタを使用
# または一回限りのリクエストスタイル、このリクエストのためにブラウザを開き、完了後に閉じる
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ page = Selector("<html>...</html>")
### 非同期 Session 管理の例
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` はコンテキストアウェアで、同期/非同期両方のパターンで動作可能
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ scrapling shell
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSS セレクタ'#fromSkipToProducts'に一致するすべての要素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_KR.md b/docs/README_KR.md
index 43d3a92..d7e03ea 100644
--- a/docs/README_KR.md
+++ b/docs/README_KR.md
@@ -56,7 +56,7 @@ Scrapling은 단일 요청부터 대규모 크롤링까지 모든 것을 처리
실시간 통계와 스트리밍을 통한 초고속 크롤링. Web Scraper가 만들고, Web Scraper와 일반 사용자 모두를 위해 설계했습니다.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # 탐지를 피해 웹사이트를 가져옵니다!
products = p.css('.product', auto_save=True) # 웹사이트 디자인 변경에도 살아남는 데이터를 스크레이핑!
@@ -243,9 +243,9 @@ MySpider().start()
### 세션을 지원하는 고급 웹사이트 가져오기
- **HTTP 요청**: `Fetcher` 클래스로 빠르고 은밀한 HTTP 요청. 브라우저의 TLS fingerprint, 헤더를 모방하고, HTTP/3를 사용할 수 있습니다.
-- **동적 로딩**: Playwright의 Chromium과 Google Chrome을 지원하는 `DynamicFetcher` 클래스로 완전한 브라우저 자동화를 통해 동적 웹사이트를 가져옵니다.
+- **동적 로딩**: Playwright의 Chromium과 Google Chrome을 지원하는 `StealthyFetcher` 클래스로 완전한 브라우저 자동화를 통해 동적 웹사이트를 가져옵니다.
- **안티봇 우회**: `StealthyFetcher`와 fingerprint 위장을 통한 고급 스텔스 기능. 자동화로 모든 유형의 Cloudflare Turnstile/Interstitial을 손쉽게 우회합니다.
-- **세션 관리**: `FetcherSession`, `StealthySession`, `DynamicSession` 클래스로 요청 간 쿠키와 상태를 관리하는 영속적 세션을 지원합니다.
+- **세션 관리**: `FetcherSession`, `StealthySession` 클래스로 요청 간 쿠키와 상태를 관리하는 영속적 세션을 지원합니다.
- **프록시 로테이션**: 모든 세션 타입에 대응하는 순환 또는 커스텀 전략의 내장 `ProxyRotator`와 요청별 프록시 오버라이드를 제공합니다.
- **도메인 및 광고 차단**: 브라우저 기반 Fetcher에서 특정 도메인(및 하위 도메인)으로의 요청을 차단하거나 내장 광고 차단(약 3,500개의 알려진 광고/트래커 도메인)을 활성화합니다.
- **DNS 유출 방지**: 프록시 사용 시 DNS 유출을 방지하기 위해 Cloudflare DoH를 통해 DNS 쿼리를 라우팅하는 선택적 DNS-over-HTTPS 지원.
@@ -259,7 +259,7 @@ MySpider().start()
- 🔍 **유사 요소 찾기**: 발견된 요소와 유사한 요소를 자동으로 찾아냅니다.
### AI 기능
-- 🤖 **MCP 서버**: AI 챗봇과 에이전트(Claude/Cursor 등)가 Scrapling을 통해 스크래핑하도록 해줍니다. 일회성 도구와 세션 기반 도구가 일반 HTTP 요청(모든 메서드), 브라우저 가져오기, Cloudflare를 우회하는 스텔스 가져오기를 모두 지원합니다. 페이지는 AI에 전달되기 전에 CSS 선택자로 범위가 좁혀지고 프롬프트 인젝션 콘텐츠가 제거되므로, 에이전트는 더 적게 읽고 더 적게 비용을 쓰며 숨겨진 텍스트에 탈취당하지 않습니다. 스크린샷, CDP를 통한 원격 브라우저, 기본적으로 안전한 HTTP 전송도 포함됩니다. ([데모 영상](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP 서버**: AI 챗봇과 에이전트(Claude/Cursor 등)가 Scrapling을 통해 스크래핑하도록 해줍니다. 일회성 도구와 세션 기반 도구가 일반 HTTP 요청(모든 메서드)과 Cloudflare를 우회하는 스텔스 브라우저 가져오기를 모두 지원합니다. 페이지는 AI에 전달되기 전에 CSS 선택자로 범위가 좁혀지고 프롬프트 인젝션 콘텐츠가 제거되므로, 에이전트는 더 적게 읽고 더 적게 비용을 쓰며 숨겨진 텍스트에 탈취당하지 않습니다. 스크린샷, CDP를 통한 원격 브라우저, 기본적으로 안전한 HTTP 전송도 포함됩니다. ([데모 영상](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: 설치만 하면 되는 [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)이 코딩 에이전트에게 라이브러리 전체를 가르쳐, 에이전트가 Scrapling으로 작성하는 코드가 추측이 아니라 현재 API에 맞도록 해줍니다.
- 📚 **RAG용 Markdown**: 한 줄(`page.markdown()`)로 어떤 페이지든 깨끗하고 정제된 LLM용 Markdown으로 변환하거나, `SiteToMarkdownSpider` 템플릿으로 웹사이트 전체를 Markdown 코퍼스로 크롤링할 수 있습니다. 과정에 LLM을 전혀 사용하지 않습니다. ([문서](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -311,14 +311,14 @@ data = page.css('#padded_content a').getall()
```
완전한 브라우저 자동화
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # 작업이 끝날 때까지 브라우저를 열어둡니다
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # 작업이 끝날 때까지 브라우저를 열어둡니다
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # 원하시면 XPath selector도 사용 가능
# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -424,7 +424,7 @@ page = Selector("<html>...</html>")
### 비동기 세션 관리 예시
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession`은 컨텍스트 인식이 가능하며 동기/비동기 패턴 모두에서 작동
page1 = session.get('https://quotes.toscrape.com/')
@@ -459,7 +459,7 @@ scrapling shell
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSS selector '#fromSkipToProducts'에 매칭되는 모든 요소
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.md
index fcc27dd..49edd3a 100644
--- a/docs/README_PT_BR.md
+++ b/docs/README_PT_BR.md
@@ -58,7 +58,7 @@ Seu parser aprende com as mudanças nos sites e relocaliza automaticamente seus
Crawls extremamente rápidos com estatísticas em tempo real e streaming. Feito por Web Scrapers para Web Scrapers e usuários comuns, há algo para todo mundo.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Busque o site sem chamar atenção!
products = p.css('.product', auto_save=True) # Extraia dados que sobrevivem a mudanças no design do site!
@@ -245,9 +245,9 @@ MySpider().start()
### Busca Avançada de Sites com Suporte a Sessões
- **Requisições HTTP**: Requisições HTTP rápidas e furtivas com a classe `Fetcher`. Pode imitar fingerprint TLS de navegadores, cabeçalhos e usar HTTP/3.
-- **Carregamento Dinâmico**: Busque sites dinâmicos com automação completa de navegador através da classe `DynamicFetcher`, compatível com o Chromium do Playwright e o Google Chrome.
+- **Carregamento Dinâmico**: Busque sites dinâmicos com automação completa de navegador através da classe `StealthyFetcher`, compatível com o Chromium do Playwright e o Google Chrome.
- **Bypass Anti-Bot**: Capacidades avançadas de stealth com `StealthyFetcher` e spoofing de fingerprint. Pode contornar facilmente todos os tipos de Turnstile/Interstitial do Cloudflare com automação.
-- **Gerenciamento de Sessão**: Suporte a sessões persistentes com as classes `FetcherSession`, `StealthySession` e `DynamicSession` para gerenciar cookies e estado entre requisições.
+- **Gerenciamento de Sessão**: Suporte a sessões persistentes com as classes `FetcherSession` e `StealthySession` para gerenciar cookies e estado entre requisições.
- **Rotação de Proxy**: `ProxyRotator` nativo com estratégias cíclicas ou personalizadas em todos os tipos de sessão, além de sobrescritas de proxy por requisição.
- **Bloqueio de Domínios e Anúncios**: Bloqueie requisições para domínios específicos (e seus subdomínios) ou habilite o bloqueio nativo de anúncios (~3.500 domínios conhecidos de anúncios/rastreadores) nos fetchers baseados em navegador.
- **Prevenção de Vazamento de DNS**: Suporte opcional a DNS-over-HTTPS para rotear consultas DNS através do DoH da Cloudflare, evitando vazamentos de DNS ao usar proxies.
@@ -261,7 +261,7 @@ MySpider().start()
- 🔍 **Encontrar Elementos Semelhantes**: Localize automaticamente elementos parecidos com os elementos encontrados.
### Recursos de IA
-- 🤖 **Servidor MCP**: Permita que chatbots e agentes de IA (Claude/Cursor/etc) façam scraping através do Scrapling com ferramentas de uso único ou baseadas em sessão, cobrindo requisições HTTP simples (qualquer método), carregamentos com navegador e carregamentos furtivos que contornam o Cloudflare. As páginas são reduzidas com seletores CSS e limpas de conteúdo de injeção de prompt antes que a IA as veja, então o agente lê menos, custa menos e não pode ser sequestrado por texto oculto. Screenshots, navegadores remotos via CDP e um transporte HTTP seguro por padrão estão incluídos. ([vídeo demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Servidor MCP**: Permita que chatbots e agentes de IA (Claude/Cursor/etc) façam scraping através do Scrapling com ferramentas de uso único ou baseadas em sessão, cobrindo requisições HTTP simples (qualquer método) e carregamentos com navegador furtivo que contornam o Cloudflare. As páginas são reduzidas com seletores CSS e limpas de conteúdo de injeção de prompt antes que a IA as veja, então o agente lê menos, custa menos e não pode ser sequestrado por texto oculto. Screenshots, navegadores remotos via CDP e um transporte HTTP seguro por padrão estão incluídos. ([vídeo demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Um [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) pronto para instalar que ensina a biblioteca inteira aos agentes de codificação, para que o código que eles escrevem com Scrapling corresponda à API atual em vez de ser adivinhado.
- 📚 **Markdown pronto para RAG**: Transforme qualquer página em Markdown limpo, sanitizado e pronto para LLMs com uma linha (`page.markdown()`), ou rastreie um site inteiro em um corpus de Markdown com o template `SiteToMarkdownSpider`, tudo sem nenhum LLM no processo. ([documentação](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -313,14 +313,14 @@ data = page.css('#padded_content a').getall()
```
Automação completa de navegador
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Mantenha o navegador aberto até terminar
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Mantenha o navegador aberto até terminar
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # Se preferir, use seletor XPath
# Ou use o estilo de requisição avulsa, ele abre o navegador para esta requisição e o fecha ao finalizar
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -426,7 +426,7 @@ E ele funciona exatamente da mesma maneira!
### Exemplos de Gerenciamento de Sessão Assíncrona
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` entende o contexto e funciona tanto em padrões sync quanto async
page1 = session.get('https://quotes.toscrape.com/')
@@ -461,7 +461,7 @@ Extraia páginas diretamente para um arquivo sem programar (por padrão, extrai
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos os elementos que correspondem ao seletor CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/README_RU.md b/docs/README_RU.md
index c792b59..163f0ae 100644
--- a/docs/README_RU.md
+++ b/docs/README_RU.md
@@ -56,7 +56,7 @@ Scrapling - это адаптивный фреймворк для Web Scraping,
Молниеносно быстрые обходы с отслеживанием статистики в реальном времени и Streaming. Создано веб-скраперами для веб-скраперов и обычных пользователей - здесь есть что-то для каждого.
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Загрузите сайт незаметно!
products = p.css('.product', auto_save=True) # Скрапьте данные, которые переживут изменения дизайна сайта!
@@ -246,9 +246,9 @@ MySpider().start()
### Продвинутая загрузка сайтов с поддержкой Session
- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS fingerprint браузера, заголовки и использовать HTTP/3.
-- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `DynamicFetcher`, поддерживающий Chromium от Playwright и Google Chrome.
+- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `StealthyFetcher`, поддерживающий Chromium от Playwright и Google Chrome.
- **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher` и подмену fingerprint'ов. Может легко обойти все типы Cloudflare Turnstile/Interstitial с помощью автоматизации.
-- **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession`, `StealthySession` и `DynamicSession` для управления cookie и состоянием между запросами.
+- **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession` и `StealthySession` для управления cookie и состоянием между запросами.
- **Ротация Proxy**: Встроенный `ProxyRotator` с циклической или пользовательскими стратегиями для всех типов сессий, а также переопределение Proxy для каждого запроса.
- **Блокировка доменов и рекламы**: Блокируйте запросы к определённым доменам (и их поддоменам) или включите встроенную блокировку рекламы (~3 500 известных рекламных/трекерных доменов) в браузерных Fetcher'ах.
- **Защита от утечки DNS**: Опциональная поддержка DNS-over-HTTPS для маршрутизации DNS-запросов через Cloudflare DoH, предотвращая утечку DNS при использовании прокси.
@@ -262,7 +262,7 @@ MySpider().start()
- 🔍 **Поиск похожих элементов**: Автоматически находите элементы, похожие на найденные.
### Возможности для ИИ
-- 🤖 **MCP-сервер**: Позвольте ИИ-чатботам и агентам (Claude/Cursor/и т.д.) скрапить через Scrapling с помощью одноразовых или сессионных инструментов, покрывающих обычные HTTP-запросы (любой метод), загрузку через браузер и скрытую загрузку в обход Cloudflare. Страницы сужаются CSS-селекторами и очищаются от prompt-injection-контента до того, как их увидит ИИ, поэтому агент читает меньше, стоит дешевле и не может быть перехвачен скрытым текстом. Скриншоты, удалённые браузеры по CDP и безопасный по умолчанию HTTP-транспорт включены. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP-сервер**: Позвольте ИИ-чатботам и агентам (Claude/Cursor/и т.д.) скрапить через Scrapling с помощью одноразовых или сессионных инструментов, покрывающих обычные HTTP-запросы (любой метод) и загрузку через скрытый браузер в обход Cloudflare. Страницы сужаются CSS-селекторами и очищаются от prompt-injection-контента до того, как их увидит ИИ, поэтому агент читает меньше, стоит дешевле и не может быть перехвачен скрытым текстом. Скриншоты, удалённые браузеры по CDP и безопасный по умолчанию HTTP-транспорт включены. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Готовый к установке [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), который обучает кодовых агентов всей библиотеке, чтобы код, который они пишут с Scrapling, соответствовал актуальному API, а не догадкам.
- 📚 **Markdown для RAG**: Превратите любую страницу в чистый, очищенный и готовый для LLM Markdown одной строкой (`page.markdown()`) или обойдите весь сайт, собрав корпус Markdown с помощью шаблона `SiteToMarkdownSpider`, и всё это без участия LLM в процессе. ([документация](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -314,14 +314,14 @@ data = page.css('#padded_content a').getall()
```
Полная автоматизация браузера
```python
-from scrapling.fetchers import DynamicFetcher, DynamicSession
+from scrapling.fetchers import StealthyFetcher, StealthySession
-with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Держите браузер открытым, пока не закончите
+with StealthySession(headless=True, disable_resources=False, network_idle=True) as session: # Держите браузер открытым, пока не закончите
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
data = page.xpath('//span[@class="text"]/text()').getall() # XPath-селектор, если вы предпочитаете его
# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
@@ -427,7 +427,7 @@ page = Selector("<html>...</html>")
### Примеры async Session
```python
import asyncio
-from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
async with FetcherSession(http3=True) as session: # `FetcherSession` контекстно-осведомлён и может работать как в sync, так и в async-режимах
page1 = session.get('https://quotes.toscrape.com/')
@@ -462,7 +462,7 @@ scrapling shell
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Все элементы, соответствующие CSS-селектору '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
-scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
+scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
> [!NOTE]
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index f2828c7..b88da11 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -64,11 +64,11 @@ State files are stored on the machine running the MCP server. They do not includ
Response bodies are read and saved only for supported text `Content-Type` values, such as HTML, JSON, XML, or JavaScript. Binary, missing, or unrecognized content types keep request metadata and headers without reading or saving their response bodies. The browser still loads resources normally.
-Only completed requests are saved, including HTTP 4xx and 5xx responses. Failed and unfinished requests are omitted. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from Playwright and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback.
+Only completed requests are saved, including HTTP 4xx and 5xx responses. Failed and unfinished requests are omitted. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from the Browser and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback.
### Shadow DOM
-Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
+Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](../fetching/stealthy.md#shadow-dom) for selector examples and limits.
### Key Capabilities
- **Smart Content Extraction**: Convert web pages/elements to Markdown, HTML, or extract a clean version of the text content
diff --git a/docs/api-reference/fetchers.md b/docs/api-reference/fetchers.md
index cd3d90c..922bd15 100644
--- a/docs/api-reference/fetchers.md
+++ b/docs/api-reference/fetchers.md
@@ -11,8 +11,8 @@ You can import all of them directly like below:
```python
from scrapling.fetchers import (
- Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher,
- FetcherSession, AsyncStealthySession, StealthySession, DynamicSession, AsyncDynamicSession
+ Fetcher, AsyncFetcher, StealthyFetcher,
+ FetcherSession, AsyncStealthySession, StealthySession
)
```
@@ -24,10 +24,6 @@ from scrapling.fetchers import (
handler: python
:docstring:
-## ::: scrapling.fetchers.DynamicFetcher
- handler: python
- :docstring:
-
## ::: scrapling.fetchers.StealthyFetcher
handler: python
:docstring:
@@ -50,14 +46,3 @@ from scrapling.fetchers import (
## ::: scrapling.fetchers.AsyncStealthySession
handler: python
:docstring:
-
-### Dynamic Sessions
-
-## ::: scrapling.fetchers.DynamicSession
- handler: python
- :docstring:
-
-## ::: scrapling.fetchers.AsyncDynamicSession
- handler: python
- :docstring:
-
diff --git a/docs/cli/extract-commands.md b/docs/cli/extract-commands.md
index 9f48a23..fa75b4a 100644
--- a/docs/cli/extract-commands.md
+++ b/docs/cli/extract-commands.md
@@ -9,7 +9,7 @@ The `scrapling extract` command lets you download and extract content from websi
1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
- 4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md), [Dynamic websites](../fetching/dynamic.md), or [Dynamic websites with hard protections](../fetching/stealthy.md).
+ 4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md) or [browser fetching](../fetching/stealthy.md).
## What is the Extract Command group?
@@ -24,7 +24,7 @@ The extract command is a set of simple terminal tools that:
!!! tip "AI-Targeted Mode"
- All extract commands support an `--ai-targeted` flag. When enabled, it extracts only the main body content, strips noise tags (script, style, noscript, svg), removes hidden elements that could be used for prompt injection (CSS-hidden, aria-hidden, template tags), strips zero-width unicode characters, and removes HTML comments. For browser commands (`fetch`/`stealthy-fetch`), it also automatically enables ad blocking. This is ideal when the output is destined for an AI model.
+ All extract commands support an `--ai-targeted` flag. When enabled, it extracts only the main body content, strips noise tags (script, style, noscript, svg), removes hidden elements that could be used for prompt injection (CSS-hidden, aria-hidden, template tags), strips zero-width unicode characters, and removes HTML comments. For the browser command (`fetch`), it also automatically enables ad blocking. This is ideal when the output is destined for an AI model.
## Quick Start
@@ -73,8 +73,7 @@ Commands:
post Perform a POST request and save the content to a file.
put Perform a PUT request and save the content to a file.
delete Perform a DELETE request and save the content to a file.
- fetch Use DynamicFetcher to fetch content with browser...
- stealthy-fetch Use StealthyFetcher to fetch content with advanced...
+ fetch Use StealthyFetcher to fetch content with advanced stealth features.
```
We will go through each command in detail below.
@@ -250,119 +249,61 @@ We will go through each command in detail below.
--help Show this message and exit.
```
-### Browsers fetching
+### Browser fetching
-Both `fetch` and `stealthy-fetch` support `--pierce-shadow` to include open Shadow DOM content. It is off by default; use `--no-pierce-shadow` to disable it explicitly. See [Shadow DOM](../fetching/dynamic.md#shadow-dom) for selector examples and limits.
+Use `fetch` for JavaScript pages, browser automation, and protected sites. It runs `StealthyFetcher` with all stealth options available.
+
+```bash
+scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
+scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
+scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
+scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
+scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
+```
+
+Use `--pierce-shadow` to include open Shadow DOM content. It is off by default; `--no-pierce-shadow` disables it explicitly. See [Shadow DOM](../fetching/stealthy.md#shadow-dom) for selector examples and limits.
```bash
scrapling extract fetch "https://example.com" content.md --pierce-shadow
-scrapling extract stealthy-fetch "https://example.com" content.md --pierce-shadow
```
-1. **fetch - Handle Dynamic Content**
+Run `scrapling extract fetch --help` to see all options:
- For websites that load content with dynamic content or have slight protection
-
- ```bash
- scrapling extract fetch [URL] [OUTPUT_FILE] [OPTIONS]
- ```
-
- **Examples:**
- ```bash
- # Wait for JavaScript to load content and finish network activity
- scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
-
- # Wait for specific content to appear
- scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
-
- # Run in visible browser mode (helpful for debugging)
- scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
- ```
- Get the available options for the command with `scrapling extract fetch --help` as follows:
- ```bash
- Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
-
- Use DynamicFetcher to fetch content with browser automation.
-
- The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
-
- Options:
- --headless / --no-headless Run browser in headless mode (default: True)
- --disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
- --pierce-shadow / --no-pierce-shadow Include open Shadow DOM content (default: False)
- --network-idle / --no-network-idle Wait for network idle (default: False)
- --timeout INTEGER Timeout in milliseconds (default: 30000)
- --wait INTEGER Additional wait time in milliseconds after page load (default: 0)
- -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
- --wait-selector TEXT CSS selector to wait for before proceeding
- --locale TEXT Specify user locale. Defaults to the system default locale.
- --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
- --proxy TEXT Proxy URL in format "http://username:password@host:port"
- -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
- --dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
- --block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
- --executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
- --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
- --help Show this message and exit.
- ```
+```text
+Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
-2. **stealthy-fetch - Bypass Protection**
+ Use StealthyFetcher to fetch content with advanced stealth features.
- For websites with anti-bot protection or Cloudflare protection
-
- ```bash
- scrapling extract stealthy-fetch [URL] [OUTPUT_FILE] [OPTIONS]
- ```
-
- **Examples:**
- ```bash
- # Bypass basic protection
- scrapling extract stealthy-fetch "https://scrapling.requestcatcher.com" content.md
-
- # Solve Cloudflare challenges
- scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
-
- # Use a proxy for anonymity.
- scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
- ```
- Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
- ```bash
- Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
-
- Use StealthyFetcher to fetch content with advanced stealth features.
-
- The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
-
- Options:
- --headless / --no-headless Run browser in headless mode (default: True)
- --disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
- --block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
- --solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
- --allow-webgl / --block-webgl Allow WebGL (default: True)
- --pierce-shadow / --no-pierce-shadow Include open Shadow DOM content (default: False)
- --network-idle / --no-network-idle Wait for network idle (default: False)
- --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
- --timeout INTEGER Timeout in milliseconds (default: 30000)
- --wait INTEGER Additional wait time in milliseconds after page load (default: 0)
- -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
- --wait-selector TEXT CSS selector to wait for before proceeding
- --hide-canvas / --show-canvas Add noise to canvas operations (default: False)
- --proxy TEXT Proxy URL in format "http://username:password@host:port"
- -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
- --dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
- --block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
- --executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
- --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
- --help Show this message and exit.
- ```
+Options:
+ --headless / --no-headless Run browser in headless mode (default: True)
+ --disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
+ --block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
+ --solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
+ --allow-webgl / --block-webgl Allow WebGL (default: True)
+ --pierce-shadow / --no-pierce-shadow Include open Shadow DOM content (default: False)
+ --network-idle / --no-network-idle Wait for network idle (default: False)
+ --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
+ --timeout INTEGER Timeout in milliseconds (default: 30000)
+ --wait INTEGER Additional wait time in milliseconds after page load (default: 0)
+ -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
+ --wait-selector TEXT CSS selector to wait for before proceeding
+ --hide-canvas / --show-canvas Add noise to canvas operations (default: False)
+ --locale TEXT Specify user locale. Defaults to the system default locale.
+ --proxy TEXT Proxy URL in format "http://username:password@host:port"
+ -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
+ --dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
+ --block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
+ --executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
+ --ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
+ --help Show this message and exit.
+```
## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
- Use **`get`** with simple websites, blogs, or news articles
-- Use **`fetch`** with modern web apps, or sites with dynamic content
-- Use **`stealthy-fetch`** with protected sites, Cloudflare, or anti-bot systems
+- Use **`fetch`** with JavaScript pages, browser automation, or protected sites, including Cloudflare
## Legal and Ethical Considerations
diff --git a/docs/cli/interactive-shell.md b/docs/cli/interactive-shell.md
index 297aa35..d0c99ae 100644
--- a/docs/cli/interactive-shell.md
+++ b/docs/cli/interactive-shell.md
@@ -11,7 +11,7 @@ The Scrapling Interactive Shell is an enhanced IPython-based environment designe
1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
- 4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md), [Dynamic websites](../fetching/dynamic.md), or [Dynamic websites with hard protections](../fetching/stealthy.md).
+ 4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md) or [browser fetching](../fetching/stealthy.md).
## Why use the Interactive Shell?
@@ -71,10 +71,9 @@ The shell provides convenient shortcuts that eliminate boilerplate code:
- **`post(url, **kwargs)`** - HTTP POST request (instead of `Fetcher.post`)
- **`put(url, **kwargs)`** - HTTP PUT request (instead of `Fetcher.put`)
- **`delete(url, **kwargs)`** - HTTP DELETE request (instead of `Fetcher.delete`)
-- **`fetch(url, **kwargs)`** - Browser-based fetch (instead of `DynamicFetcher.fetch`)
-- **`stealthy_fetch(url, **kwargs)`** - Stealthy browser fetch (instead of `StealthyFetcher.fetch`)
+- **`fetch(url, **kwargs)`** - Stealthy browser fetch (instead of `StealthyFetcher.fetch`)
-The most commonly used classes are automatically available without any import, including `Fetcher`, `AsyncFetcher`, `DynamicFetcher`, `StealthyFetcher`, and `Selector`.
+The most commonly used classes are automatically available without any import, including `Fetcher`, `AsyncFetcher`, `StealthyFetcher`, and `Selector`.
### Smart Page Management
diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md
index 33f339b..242b25a 100644
--- a/docs/fetching/choosing.md
+++ b/docs/fetching/choosing.md
@@ -9,29 +9,29 @@ This feature was introduced because, before v0.2, Scrapling was only a parsing e
## Fetchers Overview
-Scrapling provides three different fetcher classes with their session classes; each fetcher is designed for a specific use case.
+Scrapling provides HTTP fetchers (`Fetcher` and `AsyncFetcher`) and a browser fetcher (`StealthyFetcher`), with session classes for repeated requests.
The following table compares them and can be quickly used for guidance.
-| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
-|--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
-| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
-| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| JavaScript loading | ❌ | ✅ | ✅ |
-| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
-| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Small-Mid protections | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
-| Browser(s) | ❌ | Chromium and Google Chrome | Chromium and Google Chrome |
-| Browser API used | ❌ | PlayWright | PlayWright |
-| Setup Complexity | Simple | Simple | Simple |
+| Feature | Fetcher | StealthyFetcher |
+|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------------------|
+| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 |
+| Stealth | ⭐⭐ | ⭐⭐⭐⭐⭐ |
+| Anti-Bot options | ⭐⭐ | ⭐⭐⭐⭐⭐ |
+| JavaScript loading | ❌ | ✅ |
+| Memory Usage | ⭐ | ⭐⭐⭐ |
+| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
+| Browser(s) | ❌ | Chromium and Google Chrome |
+| Browser API used | ❌ | Playwright |
+| Setup Complexity | Simple | Simple |
In the following pages, we will talk about each one in detail.
## Parser configuration in all fetchers
All fetchers share the same import method, as you will see in the upcoming pages
```python
-from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
```
Then you use it right away without initializing like this, and it will use the default parser settings:
```python
@@ -78,7 +78,7 @@ page.body # Raw response body as bytes
page.encoding # Response encoding
page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
```
-All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](dynamic.md#network-history) for saved API responses and the v5 migration.
+All fetchers return the `Response` object. Browser sessions with `record_requests=True` also return saved `Response` objects from `session.network.search(limit=None)` and `session.network.get(request_id)`. See [Network History](stealthy.md#network-history) for saved API responses and the v5 migration.
The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:
```python
diff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.md
deleted file mode 100644
index 0a0706d..0000000
--- a/docs/fetching/dynamic.md
+++ /dev/null
@@ -1,490 +0,0 @@
-# Fetching dynamic websites
-
-Here, we will discuss the `DynamicFetcher` class (formerly `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
-
-As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
-
-!!! success "Prerequisites"
-
- 1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
- 2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
- 3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
-
-## Basic Usage
-You have one primary way to import this Fetcher, which is the same for all fetchers.
-
-```python
-from scrapling.fetchers import DynamicFetcher
-```
-Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
-
-Now, we will review most of the arguments one by one, using examples. If you want to jump to a table of all arguments for quick reference, [click here](#full-list-of-arguments)
-
-!!! abstract
-
- The async version of the `fetch` method is `async_fetch`, of course.
-
-
-This fetcher currently provides three main run options that can be combined as desired.
-
-Which are:
-
-### 1. Vanilla Playwright
-```python
-DynamicFetcher.fetch('https://example.com')
-```
-Using it in that manner will open a Chromium browser and load the page. There are optimizations for speed, and some stealth goes automatically under the hood, but other than that, there are no tricks or extra features unless you enable some; it's just a plain PlayWright API.
-
-### 2. Real Chrome
-```python
-DynamicFetcher.fetch('https://example.com', real_chrome=True)
-```
-If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
-
-If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
-```commandline
-playwright install chrome
-```
-
-### 3. CDP Connection
-```python
-DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
-```
-Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
-
-The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser (ex: `http://localhost:9222`).
-
-
-!!! note "Notes:"
-
- * There was a `stealth` option here, but it was moved to the `StealthyFetcher` class, as explained on the next page, with additional features since version 0.3.13.<br/>
- * This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](../fetching/stealthy.md).
-
-## Full list of arguments
-Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of how to use most of them.
-
-| Argument | Description | Optional |
-|:---------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
-| url | Target url | ❌ |
-| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
-| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
-| cookies | Set cookies for the next request. | ✔️ |
-| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
-| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
-| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
-| pierce_shadow | Include open Shadow DOM content in the response HTML. Defaults to `False`. Can be set for the session or overridden for each request. | ✔️ |
-| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
-| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
-| page_action | Added for automation. Pass a function that takes the `page` object, runs after navigation, and does the necessary automation. | ✔️ |
-| page_setup | A function that takes the `page` object, runs before navigation. Use it to register event listeners or routes that must be set up before the page loads. | ✔️ |
-| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
-| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
-| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
-| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
-| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
-| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
-| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
-| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
-| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
-| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
-| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
-| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
-| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
-| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
-| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
-| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
-| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
-| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
-| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
-| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
-| record_requests | Keep completed requests and saved responses in `session.network`. Session-only; defaults to `False`. | ✔️ |
-| max_recorded_requests | Maximum retained request entries per session; defaults to 1,000. Saved bodies also have fixed size limits. | ✔️ |
-| executable_path | Absolute path to a custom browser executable to use instead of the bundled Chromium. Useful for non-standard installations or custom browser builds. | ✔️ |
-
-In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `page_setup`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `pierce_shadow`, `blocked_domains`, `proxy`, and `selector_config`.
-
-!!! note "Notes:"
-
- 1. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
- 2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
- 3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
- 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
-
-
-## Session Management
-
-To keep the browser open until you make multiple requests with the same configuration, use `DynamicSession`/`AsyncDynamicSession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-# Create a session with default configuration
-with DynamicSession(
- headless=True,
- disable_resources=True,
- real_chrome=True
-) as session:
- # Make multiple requests with the same browser instance
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
- page3 = session.fetch('https://dynamic-site.com')
-
- # All requests reuse the same tab on the same browser instance
-```
-
-### Async Session Usage
-
-```python
-import asyncio
-from scrapling.fetchers import AsyncDynamicSession
-
-async def scrape_multiple_sites():
- async with AsyncDynamicSession(
- network_idle=True,
- timeout=30000,
- max_pages=3
- ) as session:
- # Make async requests with shared browser configuration
- pages = await asyncio.gather(
- session.fetch('https://spa-app1.com'),
- session.fetch('https://spa-app2.com'),
- session.fetch('https://dynamic-content.com')
- )
- return pages
-```
-
-You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
-
-1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
-2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
-3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
-
-Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
-
-This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
-
-Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
-
-Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
-
-### Save and Load Session State
-
-Use `save_state(path)` to save cookies, local storage and IndexedDB to a JSON file. Use `load_state(path)` to restore that file in another session, or replace the current session's stored state. Both methods accept a string or `pathlib.Path` and return `None`. The format is compatible with Playwright's storage state files.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession() as session:
- session.fetch('https://example.com')
- session.save_state('state.json')
-
-with DynamicSession() as session:
- session.load_state('state.json')
- page = session.fetch('https://example.com')
-```
-
-Save after your login or other browser actions finish. Call both methods inside a started session, after any pending fetches finish. Loading replaces existing cookies, local storage and IndexedDB. Fetch or reload the page to use the restored login; loading does not navigate existing tabs.
-
-The same methods work with `StealthySession`. Use `await` with `AsyncDynamicSession` and `AsyncStealthySession`:
-
-```python
-from scrapling.fetchers import AsyncDynamicSession
-
-async def restore_session():
- async with AsyncDynamicSession() as session:
- await session.load_state('state.json')
- page = await session.fetch('https://example.com')
- await session.save_state('state.json')
- return page
-```
-
-These methods save and restore only the shared browser context. They do not affect temporary per-request proxy contexts. Sessions with `proxy_rotator` have no shared context, so these methods raise `RuntimeError`. A fixed session `proxy` works as usual.
-
-The file does not contain `sessionStorage`, open tabs or browser settings. Saving overwrites the file; its parent directory must exist. File and JSON errors are raised to the caller.
-
-### Session Benefits
-
-- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
-- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
-- **Consistent fingerprint**: Same browser fingerprint across all requests.
-- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
-
-
-## Examples
-It's easier to understand with examples, so let's take a look.
-
-### Resource Control
-
-```python
-# Disable unnecessary resources
-page = DynamicFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
-```
-
-### Shadow DOM
-
-Some pages keep content inside open shadow roots, which are not included in the page's normal HTML. Enable `pierce_shadow` to include that content in the response, then use CSS or XPath selectors as usual:
-
-```python
-from scrapling.fetchers import DynamicFetcher
-
-page = DynamicFetcher.fetch('https://example.com', pierce_shadow=True)
-titles = page.css('product-card .title::text').getall()
-```
-
-The option defaults to `False`. You can also enable it for a session and disable it for an individual request:
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(pierce_shadow=True) as session:
- page = session.fetch('https://example.com')
- plain_page = session.fetch('https://example.com', pierce_shadow=False)
-```
-
-The same option works with `async_fetch`, async sessions, and stealth fetchers.
-
-Only open shadow roots are included. Closed roots and iframe documents are not traversed. Use `wait_selector` when a component loads after the page.
-
-Each open root becomes a `<shadow-root>` wrapper in the response. Slots keep their tags and attributes and contain their assigned nodes, or fallback content when nothing is assigned. Unassigned light children are omitted. The live page is unchanged.
-
-```html
-<product-card>
- <shadow-root>
- <div class="title">Product name</div>
- <slot name="price"><span class="price">20</span></slot>
- </shadow-root>
-</product-card>
-```
-
-Descendant selectors work across these elements. Direct-child selectors must include the wrappers and slots:
-
-```python
-titles = page.css('product-card .title::text').getall()
-titles = page.css('product-card > shadow-root > .title::text').getall()
-prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
-```
-
-The wrappers and slots appear in XPath paths, `.children`, `response.body`, and HTML exports. Markdown and text extraction return their content without printing the tags. Hidden slot attributes remain available to content cleaning.
-
-The snapshot follows content order, but it does not calculate CSS visibility or layout. Selectors containing `<shadow-root>` apply to the Scrapling response, not the live browser page. Other HTML parsers can handle unusual nesting differently.
-
-### Domain Blocking
-
-```python
-# Block requests to specific domains (and their subdomains)
-page = DynamicFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})
-```
-
-### Network Control
-
-```python
-# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
-page = DynamicFetcher.fetch('https://example.com', network_idle=True)
-
-# Custom timeout (in milliseconds)
-page = DynamicFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
-
-# Proxy support (It can also be a dictionary with only the keys 'server', 'username', and 'password'.)
-page = DynamicFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
-```
-
-### Downloading Files
-
-```python
-page = DynamicFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
-
-with open(file='main_cover.png', mode='wb') as f:
- f.write(page.body)
-```
-
-The `body` attribute of the `Response` object always returns `bytes`.
-
-### Pre-Navigation Setup
-If you need to set up event listeners, routes, or scripts that must be registered before the page navigates, use `page_setup`. This function receives the `page` object and runs before `page.goto()` is called.
-
-```python
-from playwright.sync_api import Page
-
-def capture_websockets(page: Page):
- page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
-
-page = DynamicFetcher.fetch('https://example.com', page_setup=capture_websockets)
-```
-Async version:
-```python
-from playwright.async_api import Page
-
-async def capture_websockets(page: Page):
- page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
-
-page = await DynamicFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
-```
-
-You can combine it with `page_action` -- `page_setup` runs before navigation, `page_action` runs after.
-
-### Browser Automation
-This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
-
-This function is executed immediately after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, allowing it to be used for purposes beyond automation. You can alter the page as you want.
-
-In the example below, I used the pages' [mouse events](https://playwright.dev/python/docs/api/class-mouse) to scroll the page with the mouse wheel, then move the mouse.
-```python
-from playwright.sync_api import Page
-
-def scroll_page(page: Page):
- page.mouse.wheel(10, 0)
- page.mouse.move(100, 400)
- page.mouse.up()
-
-page = DynamicFetcher.fetch('https://example.com', page_action=scroll_page)
-```
-Of course, if you use the async fetch version, the function must also be async.
-```python
-from playwright.async_api import Page
-
-async def scroll_page(page: Page):
- await page.mouse.wheel(10, 0)
- await page.mouse.move(100, 400)
- await page.mouse.up()
-
-page = await DynamicFetcher.async_fetch('https://example.com', page_action=scroll_page)
-```
-
-### Wait Conditions
-
-```python
-# Wait for the selector
-page = DynamicFetcher.fetch(
- 'https://quotes.toscrape.com/js-delayed/',
- wait_selector='.quote',
- wait_selector_state='visible'
-)
-```
-This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
-
-After that, if `load_dom` is enabled (the default), the fetcher will check again to see if all JavaScript files are loaded and executed (in the `domcontentloaded` state) or continue waiting. If you have enabled `network_idle`, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
-
-The states the fetcher can wait for can be any of the following ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
-
-- `attached`: Wait for an element to be present in the DOM.
-- `detached`: Wait for an element to not be present in the DOM.
-- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
-- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
-
-### Network History
-
-Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
-
-```python
-from scrapling.fetchers import DynamicSession
-
-with DynamicSession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com', network_idle=True)
-
-for response in session.network.search(resource_type='document', limit=None):
- print(response.meta['network_id'], response.url, response.status)
- if note := response.meta.get('body_note'):
- print(note)
- else:
- print(response.body)
- print(response.css('title::text').get())
-
-session.network.clear()
-```
-
-The same API works with `AsyncDynamicSession`, `StealthySession`, and `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-
-- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
-- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
-- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-
-Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
-
-To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
-
-The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
-
-Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-
-Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.
-
-The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.
-
-Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.
-
-By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-
-#### Migrating in v5
-
-`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
-
-### Some Stealth Features
-
-```python
-page = DynamicFetcher.fetch(
- 'https://example.com',
- google_search=True,
- useragent='Mozilla/5.0...', # Custom user agent
- locale='en-US', # Set browser locale
-)
-```
-
-### General example
-```python
-from scrapling.fetchers import DynamicFetcher
-
-def scrape_dynamic_content():
- # Use Playwright for JavaScript content
- page = DynamicFetcher.fetch(
- 'https://example.com/dynamic',
- network_idle=True,
- wait_selector='.content'
- )
-
- # Extract dynamic content
- content = page.css('.content')
-
- return {
- 'title': content.css('h1::text').get(),
- 'items': [
- item.text for item in content.css('.item')
- ]
- }
-```
-
-### Proxy Rotation
-
-```python
-from scrapling.fetchers import DynamicSession, ProxyRotator
-
-# Set up proxy rotation
-rotator = ProxyRotator([
- "http://proxy1:8080",
- "http://proxy2:8080",
- "http://proxy3:8080",
-])
-
-# Use with session - rotates proxy automatically with each request
-with DynamicSession(proxy_rotator=rotator, headless=True) as session:
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
-
- # Override rotator for a specific request
- page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
-```
-
-!!! warning
-
- Remember that by default, all browser-based fetchers and sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
-
-## When to Use
-
-Use DynamicFetcher when:
-
-- Need browser automation
-- Want multiple browser options
-- Using a real Chrome browser
-- Need custom browser config
-- Want a few stealth options
-
-If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md
index 28ab956..344d2ac 100644
--- a/docs/fetching/stealthy.md
+++ b/docs/fetching/stealthy.md
@@ -1,33 +1,65 @@
-# Fetching dynamic websites with hard protections
+# Browser fetching with StealthyFetcher
-Here, we will discuss the `StealthyFetcher` class. This class is very similar to the [DynamicFetcher](dynamic.md#introduction) class, including the browsers, the automation, and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities; most of them are handled automatically under the hood, and the rest is up to you to enable.
+The `StealthyFetcher` class loads JavaScript pages and automates Chromium or Google Chrome with anti-bot features. It uses [Playwright's API](https://playwright.dev/python/docs/intro) with stealth patches.
-As with [DynamicFetcher](dynamic.md#introduction), you will need some knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
+As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
!!! success "Prerequisites"
- 1. You've completed or read the [DynamicFetcher](dynamic.md#introduction) page since this class builds upon it, and we won't repeat the same information here for that reason.
- 2. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
- 3. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
- 4. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
+ 1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
+ 2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
+ 3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
## Basic Usage
You have one primary way to import this Fetcher, which is the same for all fetchers.
```python
- from scrapling.fetchers import StealthyFetcher
+from scrapling.fetchers import StealthyFetcher
```
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
+Now, we will review most of the arguments one by one, using examples. If you want to jump to a table of all arguments for quick reference, [click here](#full-list-of-arguments)
+
!!! abstract
The async version of the `fetch` method is `async_fetch`, of course.
+
+This fetcher currently provides three main run options that can be combined as desired.
+
+Which are:
+
+### 1. Bundled Chromium
+```python
+StealthyFetcher.fetch('https://example.com')
+```
+This opens the bundled Chromium browser with the default stealth settings and loads the page.
+
+### 2. Real Chrome
+```python
+StealthyFetcher.fetch('https://example.com', real_chrome=True)
+```
+If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
+
+If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
+```commandline
+playwright install chrome
+```
+
+### 3. CDP Connection
+```python
+StealthyFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
+```
+Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
+
+The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser (ex: `http://localhost:9222`).
+
+
## What does it do?
-The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamic.md#introduction) class, and here are some of the things it does:
+StealthyFetcher supports these anti-bot features:
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
@@ -84,33 +116,229 @@ In session classes, all these arguments can be set globally for the session. Sti
!!! note "Notes:"
- 1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
- 2. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
- 3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
- 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
- 5. `init_script` is registered with the browser context, so it runs when pages are created. Stealthy mode uses Patchright's isolated execution context by default; if your `page_action` needs to read globals that the script places on `window`, call `page.evaluate(..., isolated_context=False)` from the action.
+ 1. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
+ 2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
+ 3. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
+ 4. `init_script` is registered with the browser context, so it runs when pages are created. Stealthy mode uses Patchright's isolated execution context by default; if your `page_action` needs to read globals that the script places on `window`, call `page.evaluate(..., isolated_context=False)` from the action.
+
+## Session Management
+
+To keep the browser open until you make multiple requests with the same configuration, use `StealthySession`/`AsyncStealthySession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
+
+```python
+from scrapling.fetchers import StealthySession
+
+
+with StealthySession(
+ headless=True,
+ disable_resources=True,
+ real_chrome=True
+) as session:
+
+ page1 = session.fetch('https://example1.com')
+ page2 = session.fetch('https://example2.com')
+ page3 = session.fetch('https://dynamic-site.com')
+
+
+```
+
+### Async Session Usage
+
+```python
+import asyncio
+from scrapling.fetchers import AsyncStealthySession
+
+async def scrape_multiple_sites():
+ async with AsyncStealthySession(
+ network_idle=True,
+ timeout=30000,
+ max_pages=3
+ ) as session:
+
+ pages = await asyncio.gather(
+ session.fetch('https://spa-app1.com'),
+ session.fetch('https://spa-app2.com'),
+ session.fetch('https://dynamic-content.com')
+ )
+ return pages
+```
+
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
+
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
+
+This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
+
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
+
+### Save and Load Session State
+
+Use `save_state(path)` to save cookies, local storage and IndexedDB to a JSON file. Use `load_state(path)` to restore that file in another session, or replace the current session's stored state. Both methods accept a string or `pathlib.Path` and return `None`. The format is compatible with Playwright's storage state files.
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession() as session:
+ session.fetch('https://example.com')
+ session.save_state('state.json')
+
+with StealthySession() as session:
+ session.load_state('state.json')
+ page = session.fetch('https://example.com')
+```
+
+Save after your login or other browser actions finish. Call both methods inside a started session, after any pending fetches finish. Loading replaces existing cookies, local storage and IndexedDB. Fetch or reload the page to use the restored login; loading does not navigate existing tabs.
+
+Use `await` with `AsyncStealthySession`:
+
+```python
+from scrapling.fetchers import AsyncStealthySession
+
+async def restore_session():
+ async with AsyncStealthySession() as session:
+ await session.load_state('state.json')
+ page = await session.fetch('https://example.com')
+ await session.save_state('state.json')
+ return page
+```
+
+These methods save and restore only the shared browser context. They do not affect temporary per-request proxy contexts. Sessions with `proxy_rotator` have no shared context, so these methods raise `RuntimeError`. A fixed session `proxy` works as usual.
+
+The file does not contain `sessionStorage`, open tabs or browser settings. Saving overwrites the file; its parent directory must exist. File and JSON errors are raised to the caller.
+
+### Session Benefits
+
+- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
+- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
+- **Consistent fingerprint**: Same browser fingerprint across all requests.
+- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
+
## Examples
-It's easier to understand with examples, so we will now review most of the arguments individually. Since it's the same class as the [DynamicFetcher](dynamic.md#introduction), you can refer to that page for more examples, as we won't repeat all the examples from there.
+It's easier to understand with examples, so let's take a look.
+
+### Resource Control
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', disable_resources=True)
+```
### Shadow DOM
-Use `pierce_shadow=True` to include open Shadow DOM content in the response HTML:
+Some pages keep content inside open shadow roots, which are not included in the page's normal HTML. Enable `pierce_shadow` to include that content in the response, then use CSS or XPath selectors as usual:
```python
+from scrapling.fetchers import StealthyFetcher
+
page = StealthyFetcher.fetch('https://example.com', pierce_shadow=True)
titles = page.css('product-card .title::text').getall()
```
-The option defaults to `False` and works with sync and async fetchers. You can set it for a session or override it for each request. Each open root adds a `<shadow-root>` wrapper to the returned HTML. Slots retain their tags and attributes; include these elements in direct-child selectors. See [Shadow DOM](dynamic.md#shadow-dom) for examples and extraction limits.
+The option defaults to `False`. You can also enable it for a session and disable it for an individual request:
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession(pierce_shadow=True) as session:
+ page = session.fetch('https://example.com')
+ plain_page = session.fetch('https://example.com', pierce_shadow=False)
+```
+
+The same option works with `async_fetch` and async sessions.
+
+Only open shadow roots are included. Closed roots and iframe documents are not traversed. Use `wait_selector` when a component loads after the page.
+
+Each open root becomes a `<shadow-root>` wrapper in the response. Slots keep their tags and attributes and contain their assigned nodes, or fallback content when nothing is assigned. Unassigned light children are omitted. The live page is unchanged.
+
+```html
+<product-card>
+ <shadow-root>
+ <div class="title">Product name</div>
+ <slot name="price"><span class="price">20</span></slot>
+ </shadow-root>
+</product-card>
+```
+
+Descendant selectors work across these elements. Direct-child selectors must include the wrappers and slots:
+
+```python
+titles = page.css('product-card .title::text').getall()
+titles = page.css('product-card > shadow-root > .title::text').getall()
+prices = page.css('product-card > shadow-root > slot[name="price"] > .price::text').getall()
+```
+
+The wrappers and slots appear in XPath paths, `.children`, `response.body`, and HTML exports. Markdown and text extraction return their content without printing the tags. Hidden slot attributes remain available to content cleaning.
+
+The snapshot follows content order, but it does not calculate CSS visibility or layout. Selectors containing `<shadow-root>` apply to the Scrapling response, not the live browser page. Other HTML parsers can handle unusual nesting differently.
+
+### Domain Blocking
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', blocked_domains={"ads.example.com", "tracker.net"})
+```
+
+### Network Control
+
+```python
+
+page = StealthyFetcher.fetch('https://example.com', network_idle=True)
+
+
+page = StealthyFetcher.fetch('https://example.com', timeout=30000)
+
+
+page = StealthyFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
+```
+
+### Downloading Files
+
+```python
+page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png')
+
+with open(file='main_cover.png', mode='wb') as f:
+ f.write(page.body)
+```
+
+The `body` attribute of the `Response` object always returns `bytes`.
+
+### Pre-Navigation Setup
+If you need to set up event listeners, routes, or scripts that must be registered before the page navigates, use `page_setup`. This function receives the `page` object and runs before `page.goto()` is called.
+
+```python
+from patchright.sync_api import Page
+
+def capture_websockets(page: Page):
+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
+
+page = StealthyFetcher.fetch('https://example.com', page_setup=capture_websockets)
+```
+Async version:
+```python
+from patchright.async_api import Page
+
+async def capture_websockets(page: Page):
+ page.on("websocket", lambda ws: print(f"WebSocket opened: {ws.url}"))
+
+page = await StealthyFetcher.async_fetch('https://example.com', page_setup=capture_websockets)
+```
+
+You can combine it with `page_action` -- `page_setup` runs before navigation, `page_action` runs after.
### Cloudflare and stealth options
```python
-# Automatic Cloudflare solver
+
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
-# Works with other stealth options
+
page = StealthyFetcher.fetch(
'https://protected-site.com',
solve_cloudflare=True,
@@ -143,7 +371,7 @@ This function is executed immediately after waiting for `network_idle` (if enabl
In the example below, I used the pages' [mouse events](https://playwright.dev/python/docs/api/class-mouse) to scroll the page with the mouse wheel, then move the mouse.
```python
-from playwright.sync_api import Page
+from patchright.sync_api import Page
def scroll_page(page: Page):
page.mouse.wheel(10, 0)
@@ -154,7 +382,7 @@ page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
```
Of course, if you use the async fetch version, the function must also be async.
```python
-from playwright.async_api import Page
+from patchright.async_api import Page
async def scroll_page(page: Page):
await page.mouse.wheel(10, 0)
@@ -165,8 +393,9 @@ page = await StealthyFetcher.async_fetch('https://example.com', page_action=scro
```
### Wait Conditions
+
```python
-# Wait for the selector
+
page = StealthyFetcher.fetch(
'https://quotes.toscrape.com/js-delayed/',
wait_selector='.quote',
@@ -184,124 +413,111 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
+### Network History
+
+Enable `record_requests` on a browser session to inspect completed requests across page loads, actions, and tabs. The history covers the session's contexts, including temporary proxy contexts, and saves responses as Scrapling `Response` objects. Find API calls, HTTP errors, and redirects without sending requests again, then read saved headers and bodies even after the session closes. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included.
-### Real-world example (Amazon)
-This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Scrapling through AI
```python
-def scrape_amazon_product(url):
- # Use StealthyFetcher to bypass protection
- page = StealthyFetcher.fetch(url)
+from scrapling.fetchers import StealthySession
- # Extract product details
- return {
- 'title': page.css('#productTitle::text').get().clean(),
- 'price': page.css('.a-price .a-offscreen::text').get(),
- 'rating': page.css('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text').get(),
- 'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
- 'features': [
- li.get().clean() for li in page.css('#feature-bullets li span::text')
- ],
- 'availability': page.css('#availability')[0].get_all_text(strip=True),
- 'images': [
- img.attrib['src'] for img in page.css('#altImages img')
- ]
- }
-```
+with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
+ session.fetch('https://example.com', network_idle=True)
-## Session Management
+for response in session.network.search(resource_type='document', limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ if note := response.meta.get('body_note'):
+ print(note)
+ else:
+ print(response.body)
+ print(response.css('title::text').get())
-To keep the browser open until you make multiple requests with the same configuration, use `StealthySession`/`AsyncStealthySession` classes. Those classes can accept all the arguments that the `fetch` function can take, which enables you to specify a config for the entire session.
+session.network.clear()
+```
-```python
-from scrapling.fetchers import StealthySession
+The same API works with `AsyncStealthySession`. Recording is off by default. Set `record_requests=True` and, if needed, `max_recorded_requests` when creating the session.
-# Create a session with default configuration
-with StealthySession(
- headless=True,
- real_chrome=True,
- block_webrtc=True,
- solve_cloudflare=True
-) as session:
- # Make multiple requests with the same browser instance
- page1 = session.fetch('https://example1.com')
- page2 = session.fetch('https://example2.com')
- page3 = session.fetch('https://nopecha.com/demo/cloudflare')
-
- # All requests reuse the same tab on the same browser instance
-```
+- `session.network.search(url_pattern=None, method=None, resource_type=None, status=None, after_id=0, limit=100, include_static=True)` returns saved `Response` objects in network ID order. `limit=None` returns all retained matches. `url_pattern` is a regular expression applied when searching; invalid patterns raise an error at that point. `after_id` selects responses with a larger network ID. `include_static=False` keeps XHR/fetch requests and HTTP errors; use `True` for all resource types and saved redirects.
+- `session.network.get(request_id)` returns one saved `Response`, or `None` if it is no longer retained.
+- `session.network.clear()` removes the history. Request IDs keep increasing and are never reused.
-### Async Session Usage
+Each saved `Response` provides `.url`, `.method`, `.status`, `.headers`, `.request_headers`, `.body` bytes, `.json()`, and CSS/XPath methods directly. Its `.meta` contains `network_id`, `resource_type`, and `request_body` (bytes or `None`). Headers can be partial if full-header reads fail. These are local reads, including for async sessions; do not await them or call `.body()`. If saved content cannot be parsed, its bytes are kept, but CSS/XPath methods use an empty document.
-```python
-import asyncio
-from scrapling.fetchers import AsyncStealthySession
+To find API traffic by URL, use `session.network.search(url_pattern=r'https://api\.example\.com/.*', limit=None)`. Add `resource_type='fetch'` or `resource_type='xhr'` when you need one request type.
-async def scrape_multiple_sites():
- async with AsyncStealthySession(
- real_chrome=True,
- block_webrtc=True,
- solve_cloudflare=True,
- timeout=60000, # 60 seconds for Cloudflare challenges
- max_pages=3
- ) as session:
- # Make async requests with shared browser configuration
- pages = await asyncio.gather(
- session.fetch('https://site1.com'),
- session.fetch('https://site2.com'),
- session.fetch('https://protected-site.com')
- )
- return pages
-```
+The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
-You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
+Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
-2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
-3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+Skipped, oversized, or unreadable response bodies return `b''`, with the reason in `response.meta.get('body_note')`. Non-text bodies use `Non-text body; not saved.`. Response metadata and available headers are kept. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note.
-Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
+The history's `search`, `get`, and `clear` methods are local, synchronous calls, even on async sessions. An entry appears only after its response has been converted and saved. Browser events run while Playwright is active; async captures also need the event loop to run. Reading the history does not wait for captures.
-This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
+Closing a session does not wait for captures. Captures still running when the page navigates or the browser context closes may be omitted. Already saved responses stay readable after closure.
-Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+By default, the history keeps up to 1,000 requests. Saved response bodies have a 1 MiB limit each and a 20 MiB combined limit; old records are removed when retention limits are reached. The body limits are fixed, not session settings. These limits cover retained entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
+#### Migrating in v5
-### Network History
+`capture_xhr` and `response.captured_xhr` were removed in v5. Use an explicit browser session with `record_requests=True`, then read `session.network.search(url_pattern=..., limit=None)`. This also replaces API capture through one-shot fetchers, so keep the session instance. Filters apply to saved history across the whole session, not to capture or a single fetch. Only supported text response bodies are saved. `search()` returns at most 100 matches by default; `limit=None` returns all retained matches without changing the history's retention limits.
-Set `record_requests=True` on `StealthySession` or `AsyncStealthySession` to keep completed requests across page loads, actions, tabs, and temporary proxy contexts. Find API calls, HTTP errors, and redirects, then read saved Scrapling responses without sending requests again. Failed and unfinished requests are omitted; completed HTTP 4xx and 5xx responses are included. Saved headers and bodies remain readable after the session closes.
+### Browser Headers and Locale
```python
-from scrapling.fetchers import StealthySession
+page = StealthyFetcher.fetch(
+ 'https://example.com',
+ google_search=True,
+ useragent='Mozilla/5.0...',
+ locale='en-US',
+)
+```
-with StealthySession(record_requests=True, max_recorded_requests=1000) as session:
- session.fetch('https://example.com', network_idle=True)
+### General example
+```python
+from scrapling.fetchers import StealthyFetcher
-for response in session.network.search(resource_type='document', limit=None):
- if note := response.meta.get('body_note'):
- print(note)
- else:
- print(response.body)
+def scrape_dynamic_content():
+
+ page = StealthyFetcher.fetch(
+ 'https://example.com/dynamic',
+ network_idle=True,
+ wait_selector='.content'
+ )
+
+
+ content = page.css('.content')
+
+ return {
+ 'title': content.css('h1::text').get(),
+ 'items': [
+ item.text for item in content.css('.item')
+ ]
+ }
```
-Use `session.network.search(...)`, `.get(request_id)`, and `.clear()` to read or clear the history. `search()` and `get()` return saved `Response` objects directly. Their `.meta` contains `network_id`, `resource_type`, and `request_body`; read `.body`, `.headers`, `.request_headers`, `.status`, and `.url` on the response itself. Search returns at most 100 matches by default; `limit=None` returns all retained matches. These calls and saved response reads are local and synchronous, including on async sessions. Entries appear after conversion finishes. Reading history or closing the session does not wait for captures; captures still running at navigation or closure may be omitted.
+### Proxy Rotation
-The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
+```python
+from scrapling.fetchers import StealthySession, ProxyRotator
-Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-Recording is off by default. The default limit is 1,000 requests, with fixed limits of 1 MiB per saved response body and 20 MiB combined; old records are removed when retention limits are reached. These limits do not cover total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies. Check `response.meta.get('body_note')` for non-text, oversized, or unreadable bodies. Saved text, including empty text, has no note. HEAD responses and status codes 204, 205, and 304 remain empty with no note. Clearing the history does not reuse IDs. See [Network History](dynamic.md#network-history) for filters and saved response access. See [Migrating in v5](dynamic.md#migrating-in-v5) for the API capture changes.
+rotator = ProxyRotator([
+ "http://proxy1:8080",
+ "http://proxy2:8080",
+ "http://proxy3:8080",
+])
-### Save and Load Session State
-`StealthySession` supports `session.save_state('state.json')` and `session.load_state('state.json')`. Use `await` for both methods with `AsyncStealthySession`. The JSON file contains cookies, local storage and IndexedDB; loading replaces the current stored state. Call these methods inside a started session, after pending fetches finish. `sessionStorage` is not saved, and sessions with `proxy_rotator` are not supported. See [Save and Load Session State](dynamic.md#save-and-load-session-state) for examples and scope.
+with StealthySession(proxy_rotator=rotator, headless=True) as session:
+ page1 = session.fetch('https://example1.com')
+ page2 = session.fetch('https://example2.com')
-### Session Benefits
-- **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.
-- **Cookie persistence**: Automatic cookie and session state handling as any browser does automatically.
-- **Consistent fingerprint**: Same browser fingerprint across all requests.
-- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
+ page3 = session.fetch('https://example3.com', proxy='http://specific-proxy:8080')
+```
+
+!!! warning
+
+ Remember that by default, all browser-based fetchers and sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
## When to Use
diff --git a/docs/index.md b/docs/index.md
index d02e5f9..18a60cc 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -23,7 +23,7 @@ Its parser learns from website changes and automatically relocates your elements
Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
```python
-from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, StealthyFetcher
StealthyFetcher.adaptive = True
page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar!
products = page.css('.product', auto_save=True) # Scrape data that survives website design changes!
@@ -116,9 +116,9 @@ MySpider().start()
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
-- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium and Google's Chrome.
+- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `StealthyFetcher` class supporting Playwright's Chromium and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
-- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
+- **Session Management**: Persistent session support with `FetcherSession` and `StealthySession` classes for cookie and state management across requests.
- **Proxy Rotation**: Built-in `ProxyRotator` with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
- **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
@@ -132,7 +132,7 @@ MySpider().start()
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
### AI Features
-- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method), browser fetches, and stealth fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](ai/agent-skill.md) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `SiteToMarkdownSpider` template, all without an LLM in the loop. ([docs](ai/building-rag-systems.md))
diff --git a/docs/overview.md b/docs/overview.md
index 8953255..a782484 100644
--- a/docs/overview.md
+++ b/docs/overview.md
@@ -289,28 +289,19 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading
We have you covered if you deal with dynamic websites like most today!
-The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
+The `StealthyFetcher` class offers many options for fetching and loading web pages using Chromium-based browsers.
```python
-from scrapling.fetchers import DynamicFetcher
-page = DynamicFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
+from scrapling.fetchers import StealthyFetcher
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
print(len(page.css(".quote"))) # -> 10
# The async version of fetch
-page = await DynamicFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
+page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
print(len(page.css(".quote"))) # -> 10
```
-It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
-
-- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
-- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
-
-
-Again, this is just the tip of the iceberg with this fetcher. Check out the rest from [here](fetching/dynamic.md) for all details and the complete list of arguments.
-
-### Dynamic anti-protection loading
-We also have you covered if you deal with dynamic websites with annoying anti-protections!
+It's built on top of [Playwright](https://playwright.dev/python/) with stealth patches. Use the bundled Chromium browser, your installed Google Chrome with `real_chrome=True`, or a running browser through `cdp_url`.
-The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
+### Anti-bot features
Some of the things it does:
diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md
index 49ff15a..0be6b94 100644
--- a/docs/spiders/advanced.md
+++ b/docs/spiders/advanced.md
@@ -227,7 +227,7 @@ Key differences from `start()`:
!!! abstract
- The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results--statistics)
+ The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results-statistics)
You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed.
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md
index d6f6d65..8895cd2 100644
--- a/docs/spiders/architecture.md
+++ b/docs/spiders/architecture.md
@@ -60,7 +60,6 @@ A priority queue with built-in URL deduplication. Requests are fingerprinted bas
Manages one or more named session instances. Each session is one of:
- [FetcherSession](../fetching/static.md)
-- [AsyncDynamicSession](../fetching/dynamic.md)
- [AsyncStealthySession](../fetching/stealthy.md)
When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started with the spider start (default) or lazily (started on the first use).
diff --git a/docs/spiders/proxy-blocking.md b/docs/spiders/proxy-blocking.md
index c637592..d952317 100644
--- a/docs/spiders/proxy-blocking.md
+++ b/docs/spiders/proxy-blocking.md
@@ -43,7 +43,7 @@ Each request automatically gets the next proxy in the rotation. The proxy used i
When you use it with browser sessions, you will need some adjustments, like below:
```python
-from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, ProxyRotator
+from scrapling.fetchers import AsyncStealthySession, ProxyRotator
# String proxies work for all session types
rotator = ProxyRotator([
diff --git a/docs/spiders/sessions.md b/docs/spiders/sessions.md
index caccafd..ec8bdf8 100644
--- a/docs/spiders/sessions.md
+++ b/docs/spiders/sessions.md
@@ -3,7 +3,7 @@
!!! success "Prerequisites"
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
- 2. You're familiar with [Fetchers basics](../fetching/choosing.md) and the differences between HTTP, Dynamic, and Stealthy sessions.
+ 2. You're familiar with [Fetchers basics](../fetching/choosing.md) and the differences between HTTP and Stealthy sessions.
A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions.
@@ -17,8 +17,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md
| Session Type | Use Case |
|-------------------------------------------------|------------------------------------------|
| [FetcherSession](../fetching/static.md) | Fast HTTP requests, no JavaScript |
-| [AsyncDynamicSession](../fetching/dynamic.md) | Browser automation, JavaScript rendering |
-| [AsyncStealthySession](../fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc. |
+| [AsyncStealthySession](../fetching/stealthy.md) | JavaScript, browser automation, anti-bot bypass |
## Configuring Sessions
@@ -167,7 +166,7 @@ async def parse(self, response: Response):
Normally, when you use `FetcherSession`, `Fetcher`, or `AsyncFetcher`, you specify the HTTP method to use with the corresponding method like `.get()` and `.post()`. But while using `FetcherSession` in spiders, you can't do this. By default, the request is an _HTTP GET_ request; if you want to use another HTTP method, you have to pass it to the `method` argument, as in the above example. The reason for this is to unify the `Request` interface across all session types.
-For browser sessions (`AsyncDynamicSession`, `AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
+For browser sessions (`AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
```python
async def parse(self, response: Response):
diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md
index fe84dbb..ac3c7b3 100644
--- a/docs/tutorials/replacing_ai.md
+++ b/docs/tutorials/replacing_ai.md
@@ -59,12 +59,7 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
-It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
-
-1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) - This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
-2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) - Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
-
-We keep improving these two with each update, so stay tuned :)
+Proxies and human-like actions alone may not be enough for protected websites. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) combines browser automation with fingerprint patches and options to handle anti-bot protection. Enable `solve_cloudflare=True` for Cloudflare Turnstile and Interstitial challenges.
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data