Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs(agent): update skill with the latest changes

Karim shoair4279ca8cbc44
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index c43d0df..b8766e2 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 23f1de4..e1fc735 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -1,7 +1,7 @@
---
name: scrapling-official
description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
-version: "0.4.5"
+version: "0.4.6"
license: Complete terms in LICENSE.txt
metadata:
homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
@@ -34,13 +34,13 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo
> 2. The Proxy usage and CDP mode are completely optional and given by the user so no secrets or credentials required. Depending on the user usage.
> 3. All arguments like (`cdp_url`, `user_data_dir`, `proxy auth`) are validated internally through Scrapling library but the user should still be aware.
-**IMPORTANT**: While using the commandline scraping commands, you MUST use the commandline argument `--ai-targeted` to protect from Prompt Injection!
+**IMPORTANT**: While using the commandline scraping commands, you MUST use the commandline argument `--ai-targeted` to protect from Prompt Injection! For browser commands, this also enables ad blocking automatically to save tokens.
## Setup (once)
Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
-`pip install "scrapling[all]>=0.4.5"`
+`pip install "scrapling[all]>=0.4.6"`
Then do this to download all the browsers' dependencies:
@@ -156,7 +156,9 @@ Both (`fetch` / `stealthy-fetch`) share options:
| --wait-selector | TEXT | CSS selector to wait for before proceeding |
| --proxy | TEXT | Proxy URL in format "http://username:password@host:port" |
| -H, --extra-headers | TEXT | Extra headers in format "Key: Value" (can be used multiple times) |
-| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False) |
+| --dns-over-https / --no-dns-over-https | None | Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False) |
+| --block-ads / --no-block-ads | None | Block requests to ~3,500 known ad and tracker domains (default: False) |
+| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False). Also enables ad blocking automatically. |
This option is specific to `fetch` only:
diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md
index 344dc0f..4f645cd 100644
--- a/agent-skill/Scrapling-Skill/examples/README.md
+++ b/agent-skill/Scrapling-Skill/examples/README.md
@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
Make sure Scrapling is installed:
```bash
-pip install "scrapling[all]>=0.4.5"
+pip install "scrapling[all]>=0.4.6"
scrapling install --force
```
diff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
index 109fde6..f623521 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md
@@ -76,6 +76,8 @@ All arguments for `DynamicFetcher` and its session classes:
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
+| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
+| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
diff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
index aea8475..7f3c896 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
@@ -58,6 +58,8 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
| blocked_domains | A set of domain names to block requests to. Subdomains are also matched (e.g., `"example.com"` blocks `"sub.example.com"` too). | ✔️ |
+| block_ads | Block requests to ~3,500 known ad/tracking domains. Can be combined with `blocked_domains`. | ✔️ |
+| dns_over_https | Route DNS queries through Cloudflare's DNS-over-HTTPS to prevent DNS leaks when using proxies. | ✔️ |
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 7bbc86b..a860f0a 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -164,6 +164,10 @@ When `main_content_only=true` (the default), the server automatically sanitizes
Keep `main_content_only=true` for maximum protection.
+## Ad blocking
+
+All browser-based tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) and persistent sessions (`open_session`) automatically block requests to ~3,500 known ad and tracker domains. This is always enabled in the MCP server to save tokens and speed up page loads. No configuration needed.
+
## Setup
Start the server (stdio transport, used by most MCP clients):