Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs(agent): update the skill to catch up with all the changes

Karim shoair965415af2026
diff --git a/agent-skill/README.md b/agent-skill/README.md
index 41293dd..b88c655 100644
--- a/agent-skill/README.md
+++ b/agent-skill/README.md
@@ -1,15 +1,15 @@
# Scrapling Agent Skill
-The skill aligns with the [AgentSkill](https://agentskills.io/specification) specification, so it will be readable by [OpenClaw](https://github.com/openclaw/openclaw), [Claude Code](https://claude.com/product/claude-code), and other agentic tools. It encapsulates almost all of the documentation website's content in Markdown, so the agent doesn't have to guess anything.
+The skill follows the [AgentSkills](https://agentskills.io/specification) specification, so it's readable by [OpenClaw](https://github.com/openclaw/openclaw), [Claude Code](https://claude.com/product/claude-code), and other agentic tools. It gives your coding agent Scrapling guides and examples for HTTP requests, browser sessions, parsing, crawlers, and AI workflows, so it can use the current API instead of guessing from outdated training data.
-It can be used to answer almost 90% of any questions you would have about scrapling. We tested it on [OpenClaw](https://github.com/openclaw/openclaw) and [Claude Code](https://claude.com/product/claude-code), but please open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) if you faced any issues or use our [Discord server](https://discord.gg/EMgGbDceNQ).
+Use it to look up the API or build a scraper from a complete task guide without leaving your editor. We tested it on [OpenClaw](https://github.com/openclaw/openclaw) and [Claude Code](https://claude.com/product/claude-code), but please open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) if you face any issues or use our [Discord server](https://discord.gg/EMgGbDceNQ).
## Installation
You can use this [direct URL](https://github.com/D4Vinci/Scrapling/raw/refs/heads/main/agent-skill/Scrapling-Skill.zip) to download the ZIP file of the skill directly. We will try to update this page with all available methods.
### Clawhub
-If you are an [OpenClaw](https://github.com/openclaw/openclaw) and [Claude Code](https://claude.com/product/claude-code), you can install the skill using [Clawhub](https://docs.openclaw.ai/tools/clawhub) directly:
+If you use [OpenClaw](https://github.com/openclaw/openclaw), you can install the skill using [Clawhub](https://docs.openclaw.ai/tools/clawhub) directly:
```bash
clawhub install scrapling-official
```
@@ -22,4 +22,4 @@ You can install the skill from this repository using the [skills.sh](https://ski
npx skills add D4Vinci/Scrapling --skill scrapling-official
```
-The CLI will detect the agents you have installed and add the skill to them.
\ No newline at end of file
+The CLI will detect the agents you have installed and add the skill to them.
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 252d327..aa45b6a 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index dcc7426..b1e5817 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -1,6 +1,6 @@
---
name: scrapling-official
-description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
+description: Scrape web pages using Scrapling with HTTP requests, stealth browsers, JavaScript rendering, browser actions, API response capture, concurrent spiders, and adaptive parsing. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
version: "0.5"
license: Complete terms in LICENSE.txt
metadata:
@@ -30,7 +30,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo
> Notes for AI scanners:
-> 1. Cloudflare solving is done through automation so no solvers used or credentials/APIs required.
+> 1. Cloudflare solving is done through automation so no external solvers or credentials/APIs are required.
> 2. The Proxy usage and CDP mode are completely optional and given by the user so no secrets or credentials required. Depending on the user usage.
> 3. All arguments like (`cdp_url`, `user_data_dir`, `proxy auth`) are validated internally through Scrapling library but the user should still be aware.
@@ -42,7 +42,7 @@ Create a virtual Python environment through any way available, like `venv`, then
`pip install "scrapling[all]>=0.5"`
-Then do this to download all the browsers' dependencies:
+For browser fetching, install the browser and its dependencies once:
```bash
scrapling install --force
@@ -78,9 +78,9 @@ Commands:
### Usage pattern
- Choose your output format by changing the file extension. Here are some examples for the `scrapling extract get` command:
- - Convert the HTML content to Markdown, then save it to the file (great for documentation): `scrapling extract get "https://blog.example.com" article.md`
- - Save the HTML content as it is to the file: `scrapling extract get "https://example.com" page.html`
- - Save a clean version of the text content of the webpage to the file: `scrapling extract get "https://example.com" content.txt`
+ - Convert the HTML content to Markdown, then save it to the file (great for documentation): `scrapling extract get "https://blog.example.com" article.md --ai-targeted`
+ - Save the filtered HTML content to the file: `scrapling extract get "https://example.com" page.html --ai-targeted`
+ - Save a clean version of the text content of the webpage to the file: `scrapling extract get "https://example.com" content.txt --ai-targeted`
- Output to a temp file, read it back, then clean up.
- All commands can use CSS selectors to extract specific parts of the page through `--css-selector` or `-s`.
@@ -106,7 +106,7 @@ Those options are shared between the 4 HTTP request commands:
| --verify / --no-verify | None | Whether to verify SSL certificates (default: True) |
| --impersonate | TEXT | Browser to impersonate. Can be a single browser (e.g., Chrome) or a comma-separated list for random selection (e.g., Chrome, Firefox, Safari). |
| --stealthy-headers / --no-stealthy-headers | None | Use stealthy browser headers (default: True) |
-| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False) |
+| --ai-targeted | None | Select body content and filter common hidden elements for AI consumption (default: False) |
Options shared between `post` and `put` only:
@@ -119,22 +119,22 @@ Examples:
```bash
# Basic download
-scrapling extract get "https://news.site.com" news.md
+scrapling extract get "https://news.site.com" news.md --ai-targeted
# Download with custom timeout
-scrapling extract get "https://example.com" content.txt --timeout 60
+scrapling extract get "https://example.com" content.txt --timeout 60 --ai-targeted
# Extract only specific content using CSS selectors
-scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
+scrapling extract get "https://blog.example.com" articles.md --css-selector "article" --ai-targeted
# Send a request with cookies
-scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
+scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john" --ai-targeted
# Add user agent
-scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
+scrapling extract get "https://api.site.com" data.txt -H "User-Agent: MyBot 1.0" --ai-targeted
# Add multiple headers
-scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
+scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US" --ai-targeted
```
#### Key options (browsers)
@@ -158,10 +158,10 @@ The `fetch` command accepts these options:
| --dns-over-https / --no-dns-over-https | None | Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False) |
| --block-ads / --no-block-ads | None | Block requests to ~3,500 known ad and tracker domains (default: False) |
| --executable-path | TEXT | Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set. |
-| --ai-targeted | None | Extract only main content and sanitize hidden elements for AI consumption (default: False). Also enables ad blocking automatically. |
+| --ai-targeted | None | Select body content and filter common hidden elements for AI consumption (default: False). Also enables ad blocking automatically. |
| --locale | TEXT | Specify user locale. Defaults to the system default locale. |
| --block-webrtc / --allow-webrtc | None | Force WebRTC to respect proxy settings (default: False). |
-| --solve-cloudflare / --no-solve-cloudflare | None | Solve Cloudflare challenges (default: False). |
+| --solve-cloudflare / --no-solve-cloudflare | None | Solve Cloudflare Turnstile/Interstitial challenges (default: False). |
| --allow-webgl / --block-webgl | None | Allow WebGL (default: True). |
| --hide-canvas / --show-canvas | None | Add noise to canvas operations (default: False). |
@@ -169,23 +169,22 @@ The `fetch` command accepts these options:
Examples:
```bash
-# Wait for JavaScript to load content and finish network activity
-scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
+scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle --ai-targeted
# Wait for specific content to appear
-scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
+scrapling extract fetch "https://quotes.toscrape.com/js/" quotes.txt --wait-selector ".quote" --ai-targeted
# Run in visible browser mode (helpful for debugging)
-scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
+scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources --ai-targeted
# Bypass basic protection
-scrapling extract fetch "https://scrapling.requestcatcher.com" content.md
+scrapling extract fetch "https://scrapling.requestcatcher.com" content.md --ai-targeted
# Solve Cloudflare challenges
-scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
+scrapling extract fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a" --ai-targeted
# Use a proxy for anonymity.
-scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
+scrapling extract fetch "https://site.com" content.md --proxy "http://proxy-server:8080" --ai-targeted
```
@@ -225,7 +224,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Keep t
data = page.css('#padded_content a').getall()
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Full browser automation
@@ -281,7 +280,7 @@ result.items.to_json("quotes.json")
```
Use multiple session types in a single spider:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class MultiSessionSpider(Spider):
@@ -296,15 +295,15 @@ class MultiSessionSpider(Spider):
for link in response.css('a::attr(href)').getall():
# Route protected pages through the stealth session
if "protected" in link:
- yield Request(link, sid="stealth")
+ yield response.follow(link, sid="stealth")
else:
- yield Request(link, sid="fast", callback=self.parse) # explicit callback
+ yield response.follow(link, sid="fast", callback=self.parse) # explicit callback
```
Pause and resume long crawls with checkpoints by running the spider like this:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.
+Press Ctrl+C to pause gracefully - pending requests and seen URLs are saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.
While iterating on a spider's `parse()` logic, set `development_mode = True` on the spider class to cache responses to disk on the first run and replay them on subsequent runs - so you can re-run the spider as many times as you want without re-hitting the target servers. The cache lives in `.scrapling_cache/{spider.name}/` by default and can be overridden with `development_cache_dir`. Don't ship a spider with this enabled.
@@ -329,7 +328,7 @@ For sitemap-driven crawls, use `SitemapSpider` with the same `rules()` API. It f
For XML feeds (RSS, Atom, product feeds), use `XMLFeedSpider`: set `itertag` to the node name and override `parse_node(response, node)`, which receives each matching node as a namespace-stripped `lxml` element (`node.findtext("title")`). For CSV feeds, use `CSVFeedSpider`: override `parse_row(response, row)`, which receives each row as a dictionary, with `headers`/`delimiter`/`quotechar` for non-standard feeds. Both decompress gzipped feeds automatically. See `references/spiders/generic-templates.md`.
-For Shopify-powered stores, subclass `ShopifySpider` and set `target_website` to the store's domain; it extracts every product variant through Shopify's JSON API without touching the HTML. See `references/spiders/platform-templates.md`.
+For Shopify-powered stores, subclass `ShopifySpider` and set `target_website` to the store's domain; it extracts product variants through Shopify's public collection API without touching the HTML. See `references/spiders/platform-templates.md`.
### Advanced Parsing & Navigation
```python
@@ -347,18 +346,18 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # and so on...
# Find element by text content
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Advanced navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Chained selectors
+quote_texts = page.css('.quote').css('.text::text').getall() # Chained selectors
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
parent_container = first_quote.parent
# Element relationships and similarity
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
You can use the parser right away if you don't want to fetch websites like below:
```python
@@ -368,33 +367,38 @@ page = Selector("<html>...</html>")
```
And it works precisely the same way!
### Async Session Management Examples
+
+`FetcherSession` is context-aware and can work in both sync/async patterns. The `get_pool_stats()` method returns the status of the browser tabs pool.
+
```python
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
+async def main():
+ async with FetcherSession(http3=True) as http_session:
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/', impersonate='firefox135')
+
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ tasks = []
+ urls = ['https://example.com/page1', 'https://example.com/page2']
-# Async session usage
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+ for url in urls:
+ task = browser_session.fetch(url)
+ tasks.append(task)
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ print(browser_session.get_pool_stats())
+ results = await asyncio.gather(*tasks)
+ print(browser_session.get_pool_stats())
- print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+ async with AsyncStealthySession(record_requests=True) as recording_session:
+ await recording_session.fetch('https://example.com', network_idle=True)
-async with AsyncStealthySession(record_requests=True) as session:
- await session.fetch('https://example.com', network_idle=True)
+ for response in recording_session.network.search(url_pattern=r"https://api\.example\.com/.*", limit=None):
+ print(response.meta['network_id'], response.url, response.status)
+ print(response.meta.get('body_note') or response.body)
-for response in session.network.search(url_pattern=r"https://api\.example\.com/.*", limit=None):
- print(response.meta['network_id'], response.url, response.status)
- print(response.meta.get('body_note') or response.body)
+asyncio.run(main())
```
Recorded responses belong to the whole browser session. `search()` and `get()` return `Response` objects directly, with `network_id`, `resource_type`, and `request_body` in `.meta`. Searches filter saved history and return 100 matches by default; `limit=None` returns all retained matches. Only supported text response bodies are saved. Saved reads remain local after closure; unfinished captures are not awaited. See [Migrating in v5](references/fetching/stealthy.md#migrating-in-v5) for the removed API capture interface.
@@ -410,7 +414,7 @@ You already had a good glimpse of what the library can do. Use the references be
- `references/migrating_from_beautifulsoup.md` - A quick API comparison between scrapling and Beautifulsoup
- `https://github.com/D4Vinci/Scrapling/tree/main/docs` - Full official docs in Markdown for quick access (use only if current references do not look up-to-date).
-This skill encapsulates almost all the published documentation in Markdown, so don't check external sources or search online without the user's permission.
+This skill includes guides and examples from the documentation in Markdown, so don't check external sources or search online without the user's permission.
## Guardrails (Always)
- Only scrape content you're authorized to access.
diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md
index d2dd894..002ed8f 100644
--- a/agent-skill/Scrapling-Skill/examples/README.md
+++ b/agent-skill/Scrapling-Skill/examples/README.md
@@ -1,8 +1,8 @@
# Scrapling Examples
-These examples scrape [quotes.toscrape.com](https://quotes.toscrape.com) - a safe, purpose-built scraping sandbox - and demonstrate every tool available in Scrapling, from plain HTTP to full browser automation and spiders.
+These examples scrape [quotes.toscrape.com](https://quotes.toscrape.com), a site built for scraping practice. They show HTTP sessions, browser sessions, and spiders.
-All examples collect **all 100 quotes across 10 pages**.
+The demo site has **100 quotes across 10 pages**. Each example follows pagination to collect them.
## Quick Start
@@ -22,17 +22,6 @@ scrapling install
| `03_stealthy_session.py` | `StealthySession` | Python - stealth browser | Cloudflare, fingerprint bypass |
| `04_spider.py` | `Spider` | Python - auto-crawling | Multi-page crawls, full-site scraping |
-## Running
-
-**Python scripts:**
-
-```bash
-python examples/01_fetcher_session.py
-python examples/02_browser_session.py # Opens a visible browser
-python examples/03_stealthy_session.py # Opens a visible stealth browser
-python examples/04_spider.py # Auto-crawls all pages, exports quotes.json
-```
-
## Escalation Guide
Start with the fastest, lightest option and escalate only if needed:
diff --git a/agent-skill/Scrapling-Skill/references/building-rag-systems.md b/agent-skill/Scrapling-Skill/references/building-rag-systems.md
index 8b56783..4d83051 100644
--- a/agent-skill/Scrapling-Skill/references/building-rag-systems.md
+++ b/agent-skill/Scrapling-Skill/references/building-rag-systems.md
@@ -1,4 +1,4 @@
-# Building RAG Systems
+# Building RAG Systems with Scrapling
RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page.
@@ -14,7 +14,7 @@ The `rag` extra installs the fetchers with Markdown conversion support (the `ai`
## One page to Markdown
-Every [Response](fetching/choosing.md) has a `markdown()` method:
+Every [Response](fetching/choosing.md#response-object) has a `markdown()` method:
```python
from scrapling.fetchers import Fetcher
@@ -39,7 +39,7 @@ Two arguments control the output:
markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")
```
-Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model.
+Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: inline hidden elements, `aria-hidden="true"` elements, hidden slots, `<template>` tags, HTML comments, and zero-width characters are removed before conversion. The MCP server uses this cleaning with `main_content_only=True`.
## A whole website to a Markdown corpus
@@ -67,7 +67,7 @@ The template requires `allowed_domains` so the crawl stays bound to the target w
- `output_dir`: When set, writes one Markdown file per page.
- `max_pages`: Maximum number of pages to convert. Requests already queued when the cap hits may still be fetched, but they aren't converted. `0` (the default) disables it.
-Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):
+Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md#crawlspider), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md#linkextractor-reference) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):
```python
from scrapling.spiders import CrawlRule, LinkExtractor, ContentSpider
@@ -90,7 +90,7 @@ The JSONL output plugs into any embedding pipeline. A minimal example:
```python
import json
-with open("docs.jsonl") as f:
+with open("docs.jsonl", encoding="utf-8") as f:
for line in f:
page = json.loads(line)
for chunk in split_into_chunks(page["markdown"]):
@@ -101,4 +101,4 @@ Use `css_selector` on the spider to cut boilerplate before chunking instead of c
## Interactive alternatives
-For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools, and this agent skill teaches coding agents to write this code themselves.
+For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools.
diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
index a9f2739..bb05286 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
@@ -21,21 +21,19 @@ The following table compares them and can be quickly used for guidance.
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐⭐⭐ |
| JavaScript loading | ❌ | ✅ |
| Memory Usage | ⭐ | ⭐⭐⭐ |
-| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
+| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites - Small automation - Small-Complicated protections |
| Browser(s) | ❌ | Chromium and Google Chrome |
| Browser API used | ❌ | Playwright |
| Setup Complexity | Simple | Simple |
-In the following pages, we will talk about each one in detail.
-
## Parser configuration in all fetchers
-All fetchers share the same import method, as you will see in the upcoming pages
+All fetchers share the same import method:
```python
from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher
```
Then you use it right away without initializing like this, and it will use the default parser settings:
```python
-page = StealthyFetcher.fetch('https://example.com')
+page = StealthyFetcher.fetch('https://example.com')
```
If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
```python
diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.md
index 516b66e..fb4687f 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/static.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/static.md
@@ -3,7 +3,7 @@
The `Fetcher` class provides rapid and lightweight HTTP requests using the high-performance `curl_cffi` library with a lot of stealth capabilities.
## Basic Usage
-Import the Fetcher (same import pattern for all fetchers):
+You have one primary way to import this Fetcher, which is the same for all fetchers.
```python
from scrapling.fetchers import Fetcher
@@ -17,7 +17,7 @@ All methods for making requests here share some arguments, so let's discuss them
- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
- **follow_redirects**: Controls redirect behavior. **Defaults to `"safe"`**, which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass `True` to follow all redirects without restriction, or `False` to disable redirects entirely.
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
-- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
+- **retries**: The maximum number of attempts for failed requests, including the first attempt. **Defaults to 3**.
- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.
- **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings or a list of them like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear to come from real browsers at the TLS level. If you pass it a list of strings, it will choose a random one with each request. **Defaults to the latest available Chrome version.**
- **http3**: Use HTTP/3 protocol for requests. **Defaults to False**. It might be problematic if used with `impersonate`.
@@ -33,6 +33,7 @@ All methods for making requests here share some arguments, so let's discuss them
- **selector_config**: A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class.
**Notes:**
+
1. The currently available browsers to impersonate are (`"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`)
2. The available browsers to impersonate, along with their corresponding versions, are automatically displayed in the argument autocompletion and updated with each `curl_cffi` update.
3. If any of the arguments `impersonate` or `stealthy_headers` are enabled, the fetchers will automatically generate real browser headers that match the browser version used.
@@ -70,7 +71,7 @@ page = Fetcher.get('https://example.com', impersonate='chrome')
# HTTP/3 support
page = Fetcher.get('https://example.com', http3=True)
```
-And for asynchronous requests, it's a small adjustment
+And for asynchronous requests, it's a small adjustment
```python
from scrapling.fetchers import AsyncFetcher
# Basic GET
@@ -79,7 +80,6 @@ page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealt
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
-
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
@@ -89,11 +89,11 @@ page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
# HTTP/3 support
page = await AsyncFetcher.get('https://example.com', http3=True)
```
-The `page` object in all cases is a [Response](choosing.md#response-object) object, which is a [Selector](../parsing/main_classes.md#selector), so you can use it directly
+Needless to say, the `page` object in all cases is [Response](choosing.md#response-object) object, which is a [Selector](../parsing/main_classes.md#selector) as we said, so you can use it directly
```python
->>> page.css('.something.something')
+page.css('.something.something')
->>> page = Fetcher.get('https://api.github.com/events')
+page = Fetcher.get('https://api.github.com/events')
>>> page.json()
[{'id': '<redacted>',
'type': 'PushEvent',
@@ -286,10 +286,10 @@ from scrapling.fetchers import Fetcher
def scrape_products():
page = Fetcher.get('https://example.com/products')
-
+
# Find all product elements
products = page.css('.product')
-
+
results = []
for product in products:
results.append({
@@ -298,7 +298,7 @@ def scrape_products():
'description': product.css('.description::text').get(),
'in_stock': product.has_class('in-stock')
})
-
+
return results
```
@@ -321,26 +321,26 @@ def scrape_all_pages():
base_url = 'https://example.com/products?page={}'
page_num = 1
all_products = []
-
+
while True:
# Get current page
page = Fetcher.get(base_url.format(page_num))
-
+
# Find products
products = page.css('.product')
if not products:
break
-
+
# Process products
for product in products:
all_products.append({
'name': product.css('.name::text').get(),
'price': product.css('.price::text').get()
})
-
+
# Next page
page_num += 1
-
+
return all_products
```
@@ -372,21 +372,21 @@ from scrapling.fetchers import Fetcher
def extract_table():
page = Fetcher.get('https://example.com/data')
-
+
# Find table
table = page.css('table')[0]
-
+
# Extract headers
headers = [
th.text for th in table.css('thead th')
]
-
+
# Extract rows
rows = []
for row in table.css('tbody tr'):
cells = [td.text for td in row.css('td')]
rows.append(dict(zip(headers, cells)))
-
+
return rows
```
@@ -397,10 +397,10 @@ from scrapling.fetchers import Fetcher
def extract_menu():
page = Fetcher.get('https://example.com')
-
+
# Find navigation
nav = page.css('nav')[0]
-
+
menu = {}
for item in nav.css('li'):
links = item.css('a')
@@ -410,7 +410,7 @@ def extract_menu():
'url': link['href'],
'has_submenu': bool(item.css('.submenu'))
}
-
+
return menu
```
@@ -435,4 +435,4 @@ Use other fetchers when:
- Need browser automation.
- Need advanced anti-bot/stealth capabilities.
-- Need JavaScript support or interacting with dynamic content
\ No newline at end of file
+- Need JavaScript support or interacting with dynamic content
diff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
index 63d4279..3834b5d 100644
--- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md
@@ -2,7 +2,7 @@
The `StealthyFetcher` class loads JavaScript pages and automates Chromium or Google Chrome with anti-bot features. It uses [Playwright's API](https://playwright.dev/python/docs/intro) with stealth patches.
-As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
+To automate the page, use [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
## Basic Usage
You have one primary way to import this Fetcher, which is the same for all fetchers.
@@ -12,8 +12,6 @@ from scrapling.fetchers import StealthyFetcher
```
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
-Now, we will review most of the arguments one by one, using examples. If you want to jump to a table of all arguments for quick reference, [click here](#full-list-of-arguments)
-
**Abstract:**
The async version of the `fetch` method is `async_fetch`, of course.
@@ -53,7 +51,7 @@ The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed br
StealthyFetcher supports these anti-bot features:
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically with `solve_cloudflare=True`.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
@@ -61,7 +59,7 @@ StealthyFetcher supports these anti-bot features:
6. and other anti-protection options...
## Full list of arguments
-Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
+Scrapling provides many options with this fetcher and its session classes.
| Argument | Description | Optional |
@@ -90,7 +88,7 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
-| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
+| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. Defaults to `False`. | ✔️ |
| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
@@ -340,7 +338,7 @@ page = StealthyFetcher.fetch(
real_chrome=True,
hide_canvas=True,
google_search=True,
- proxy='http://username:password@host:port',
+ proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
)
```
diff --git a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md b/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
index da12a4a..53dad95 100644
--- a/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
+++ b/agent-skill/Scrapling-Skill/references/integrations/scrapy.md
@@ -1,8 +1,10 @@
-# Scrapy
+# Integrating Scrapling into Scrapy
If you have an existing Scrapy project, you don't need to rewrite it to enjoy Scrapling's parsing API. The Scrapy integration converts Scrapy responses to Scrapling [Response](../fetching/choosing.md#response-object) objects right inside your spider callbacks, so Scrapy keeps handling the crawling while Scrapling handles the parsing.
-**Installation:** This integration works with the default Scrapling installation (`pip install scrapling`), no extras needed. It only requires Scrapy to be installed, which you already have in a Scrapy project.
+**Installation:**
+
+This integration works with the default Scrapling installation (`pip install scrapling`), no extras needed. It only requires Scrapy to be installed, which you already have in a Scrapy project.
## Usage
diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md
index 66de6fc..3c4b991 100644
--- a/agent-skill/Scrapling-Skill/references/mcp-server.md
+++ b/agent-skill/Scrapling-Skill/references/mcp-server.md
@@ -1,445 +1,189 @@
-# Scrapling MCP Server
+# Scrapling MCP Server Guide
-The Scrapling MCP server exposes sixteen tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), plain HTTP requests and stealth browser rendering with anti-bot bypass, persistent browser session management, mouse actions, batch field filling, keyboard shortcuts, page waits, custom JavaScript, network history, and page screenshots returned as real image content blocks. Every fetch takes one URL. One-shot tools (`make_request`, `browser_fetch_once`) close their own client or browser after the call. Use sessions opened with `browser_open` or `open_request_session` for related requests, browser actions, or network history. Browser session calls reuse one tab, so fetch pages and run actions in order. Close each persistent session when done.
+The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful Web Scraping capabilities directly to your favorite AI chatbot or AI agent. This integration allows you to scrape websites, extract data, and bypass anti-bot protections conversationally through Claude's AI interface or any interface that supports MCP.
-Fetch and HTTP request tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `browser_screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the current page URL). `browser_extract` and `browser_actions` return plain text. Both network tools return structured JSON; the request list supports pagination. `browser_evaluate` returns compact JSON in one plain text block.
+## Features
-## Shadow DOM
+The Scrapling MCP Server provides sixteen tools for web scraping and browser interaction. One-shot tools fetch a single URL and close their own browser or client. Session tools keep cookies and settings across calls, so the AI can continue browser actions on the same tab and inspect network history.
-Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](fetching/stealthy.md#shadow-dom) for selector examples and limits.
+### One-shot tools
+#### 🚀 Basic HTTP Scraping
+- **`make_request`**: Fast HTTP requests with any method (GET, POST, PUT, DELETE) and browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
-## One-shot tools
-
-### `make_request` -- HTTP request, any method (single URL)
-
-Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection. Closes its own HTTP client after the call; use `session_make_request` for a persistent session.
-
-**Key parameters:**
-
-| Parameter | Type | Default | Description |
-|---------------------|-------------------------------------------|--------------|--------------------------------------------------------------------|
-| `url` | str | required | URL to fetch |
-| `method` | `"GET"` / `"POST"` / `"PUT"` / `"DELETE"` | `"GET"` | HTTP method |
-| `data` | dict or str or null | null | Request body (form data). POST/PUT/DELETE only |
-| `json` | dict or list or null | null | Request body (JSON). POST/PUT/DELETE only |
-| `extraction_type` | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format |
-| `css_selector` | str or null | null | CSS selector to narrow content (applied after `main_content_only`) |
-| `main_content_only` | bool | true | Restrict to `<body>` content |
-| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate |
-| `proxy` | str or null | null | Proxy URL, e.g. `"http://user:pass@host:port"` |
-| `proxy_auth` | dict or null | null | `{"username": "...", "password": "..."}` |
-| `auth` | dict or null | null | HTTP basic auth, same format as proxy_auth |
-| `timeout` | number | 30 | Seconds before timeout |
-| `retries` | int | 3 | Maximum attempts, including the first |
-| `retry_delay` | int | 1 | Seconds between retries |
-| `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer |
-| `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |
-| `follow_redirects` | bool or "safe" | "safe" | Follow redirects. "safe" rejects redirects to internal/private IPs |
-| `max_redirects` | int | 30 | Max redirects (-1 for unlimited) |
-| `headers` | dict or null | null | Custom request headers |
-| `cookies` | dict or null | null | Request cookies |
-| `params` | dict or null | null | Query string parameters |
-| `verify` | bool | true | Verify HTTPS certificates |
-
-### `browser_fetch_once` -- Stealth browser fetch (single URL)
-
-Uses a stealthy Chromium browser through Patchright for JavaScript rendering, fingerprint spoofing, and anti-bot bypass. Supports dynamic/SPA pages and sites with Cloudflare Turnstile/Interstitial or other strong protections.
-
-**Key parameters:**
-
-| Parameter | Type | Default | Description |
-|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|
-| `url` | str | required | URL to fetch |
-| `extraction_type` | str | `"markdown"` | `"markdown"` / `"html"` / `"text"` |
-| `css_selector` | str or null | null | Narrow content before extraction |
-| `main_content_only` | bool | true | Restrict to `<body>` |
-| `headless` | bool | true | Run browser hidden (true) or visible (false) |
-| `proxy` | str or dict or null | null | String URL or `{"server": "...", "username": "...", "password": "..."}` |
-| `timeout` | number | 30000 | Timeout in **milliseconds** |
-| `wait` | number | 0 | Extra wait (ms) after page load before extraction |
-| `wait_selector` | str or null | null | CSS selector to wait for before extraction |
-| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
-| `network_idle` | bool | false | Wait until no network activity for 500ms |
-| `pierce_shadow` | bool | false | Include open Shadow DOM content in this request. |
-| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
-| `google_search` | bool | true | Set a Google referer header |
-| `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium |
-| `cdp_url` | str or null | null | Connect to existing browser via CDP URL |
-| `extra_headers` | dict or null | null | Additional request headers |
-| `useragent` | str or null | null | Custom user-agent (auto-generated if null) |
-| `cookies` | list or null | null | Playwright-format cookies |
-| `timezone_id` | str or null | null | Browser timezone, e.g. `"America/New_York"` |
-| `locale` | str or null | null | Browser locale, e.g. `"en-GB"` |
-| `solve_cloudflare` | bool | false | Automatically solve Cloudflare Turnstile/Interstitial challenges |
-| `hide_canvas` | bool | false | Add noise to canvas operations |
-| `block_webrtc` | bool | false | Disable non-proxied WebRTC UDP to reduce IP leaks |
-| `allow_webgl` | bool | true | Keep WebGL enabled; disabling it can trigger bot detection |
-| `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |
-
-This one-shot tool creates and closes its own stealthy browser session. For related fetches, later browser actions, or network history, use `browser_open` and `browser_fetch`.
-
-## Session tools
-
-### `browser_open` -- Create a persistent stealthy browser session
-
-Opens a stealthy browser session that stays alive across multiple `browser_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `browser_fetch`. For plain HTTP requests without a browser, use `open_request_session` instead. Returns a `SessionCreatedModel` with `session_id`, `session_type="stealthy"`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.
-
-**Key parameters:**
-
-| Parameter | Type | Default | Description |
-|--------------------|-----------------------------|--------------|-------------------------------------------------------------------------------------------------------|
-| `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |
-| `headless` | bool | true | Run browser hidden or visible |
-| `hide_canvas` | bool | false | Canvas fingerprint noise |
-| `block_webrtc` | bool | false | Block WebRTC IP leak |
-| `allow_webgl` | bool | true | Keep WebGL enabled |
-
-Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `browser_fetch`.
-
-Use `solve_cloudflare` on `browser_fetch` to solve Cloudflare challenges through the session.
-
-Recording of completed requests starts automatically for this session. Use `browser_network_requests` to find traffic and `browser_network_request` to inspect a recorded request.
-
-### `open_request_session` -- Create a persistent HTTP requests session
-
-Opens an HTTP session (no browser) that stays alive across multiple `session_make_request` calls, keeping cookies, connections, and the browser fingerprint between requests. Returns the same `SessionCreatedModel` receipt and shows in `list_sessions` as a `static` session.
-
-| Parameter | Type | Default | Description |
-|---------------|-------------|------------|-------------------------------------------------------------------------------------------------------|
-| `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |
-| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate on every request |
-| `proxy` | str or null | null | Proxy URL used for every request, e.g. `"http://user:pass@host:port"` |
-
-### `browser_save_state` and `browser_load_state` -- Save or restore browser storage
-
-Save cookies, local storage and IndexedDB to a Playwright-compatible JSON file, or replace an open browser session's stored state from that file. Both tools use the session's `save_state` and `load_state` methods. They work immediately after `browser_open`; no `browser_fetch` is required. HTTP sessions are not supported.
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID from `browser_open` |
-| `path` | nonempty str | required | JSON file path on the MCP server host; relative paths use its working directory |
-
-Saving overwrites the file, and its parent directory must exist. Loading replaces cookies, local storage and IndexedDB. It leaves `sessionStorage` and open tabs unchanged; fetch or reload pages to use the restored login. The file does not save `sessionStorage`, tabs or browser settings. Finish other calls in the session before saving or loading.
+#### 🔒 Stealth Browser Scraping
+- **`browser_fetch_once`**: Render JavaScript and scrape dynamic pages with our Stealthy browser. Supports fingerprint spoofing, Cloudflare Turnstile/Interstitial bypass, and control over the browser and request!
-```json
-{"session_id": "browser", "path": "state.json"}
-```
+### Session tools
-Save returns `State saved.` and load returns `State loaded.` as plain text, with no storage values in the response. Unknown, closed or HTTP sessions, invalid paths, unreadable files and malformed JSON return errors. These tools do not open or close sessions.
+#### 🔌 Session Management
+- **`browser_open`**: Create a persistent stealthy browser session that stays open across multiple `browser_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration and returns only `session_id` and `session_type`. Use `list_sessions` to see its effective `settings` (empty for CDP sessions).
+- **`open_request_session`**: Create a persistent HTTP requests session (no browser) used with `session_make_request`, keeping cookies, connections, and the browser fingerprint (`impersonate`) between requests. It returns only `session_id` and `session_type`, with type `static`. Use `list_sessions` to see its effective `settings`.
+- **`close_session`**: Close a persistent session (browser or requests) and free its resources.
+- **`list_sessions`**: List all active sessions with their details and `settings`.
-### `browser_fetch` -- Fetch through an open browser session (single URL)
+#### Browser State
-Fetches one URL through a stealthy Chromium session opened with `browser_open`, with JavaScript rendering, fingerprint spoofing, and Cloudflare Turnstile/Interstitial bypass. The session holds the browser-level configuration; every parameter here applies to this request only. Raises on a requests session; use `session_make_request` there instead.
+- **`browser_save_state`**: Save cookies, local storage and IndexedDB to a Playwright-compatible JSON file, so login state can be reused after the browser session ends. Saving replaces an existing file.
+- **`browser_load_state`**: Restore a saved file into an open browser session, including before its first page load. This replaces the current stored state without closing or navigating existing tabs. Fetch or reload the page to use the restored login.
-| Parameter | Type | Default | Description |
-|-----------------------|--------------|--------------|----------------------------------------------------------------------------------------|
-| `url` | str | required | URL to fetch |
-| `session_id` | str | required | ID of an open session created with `browser_open` |
-| `extraction_type` | str | `"markdown"` | `"markdown"` / `"html"` / `"text"` / `"snapshot"` |
-| `css_selector` | str or null | null | Narrow content before extraction |
-| `main_content_only` | bool | true | Restrict to `<body>` |
-| `wait` | number | 0 | Extra wait (ms) after page load before extraction |
-| `timeout` | number | 30000 | Timeout in **milliseconds** |
-| `google_search` | bool | true | Set a Google referer header |
-| `network_idle` | bool | false | Wait until no network activity for 500ms |
-| `pierce_shadow` | bool | false | Include open Shadow DOM content in this request. |
-| `load_dom` | bool | true | Wait for the page's JavaScript to fully load and execute |
-| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
-| `wait_selector` | str or null | null | CSS selector to wait for before extraction |
-| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
-| `extra_headers` | dict or null | null | Additional request headers |
-| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |
-| `solve_cloudflare` | bool | false | Auto-solve Cloudflare Turnstile/Interstitial challenges |
+State files are stored on the machine running the MCP server. They do not include `sessionStorage`, open tabs or browser settings.
-With `extraction_type="snapshot"`, the response keeps `status` and `url` and returns one unchanged AI ARIA snapshot string in `content`. These snapshots always include element positions and sizes in viewport CSS pixels. `css_selector` must match exactly one element; omit it for the whole page. `main_content_only` and `pierce_shadow` do not filter snapshots. Snapshot extraction is only available on `browser_fetch`.
+#### 🎯 Fetching Through a Session
+- **`browser_fetch`**: Render JavaScript and scrape a single URL through an open stealthy browser session, with fingerprint spoofing and Cloudflare Turnstile/Interstitial bypass. It can also capture a structured page snapshot with element references, positions, and sizes, helping the AI identify controls and plan mouse actions.
+- **`session_make_request`**: Make fast HTTP requests with any method (GET, POST, PUT, DELETE), browser fingerprint impersonation, matching browser headers, and HTTP/3 support. Reuse cookies, connections, and the browser fingerprint through a session opened with `open_request_session`.
-### `browser_extract` -- Read current page content without navigation
+#### 📸 Screenshots
-Returns one plain-text block from the current browser page. `extraction_type` defaults to `"snapshot"`; use `"html"`, `"markdown"`, or `"text"` for content extraction after page actions. Call `browser_fetch` first. Unknown or HTTP sessions and missing, closed, or busy pages return an error. The page stays reserved during capture and conversion and is released after success, failure, or cancellation.
+- **`browser_screenshot`**: Capture the current page or a chosen element as PNG or JPEG without reloading it. Inspect filled fields, open menus, charts, or result cards, with less unrelated content in element captures. Supports the visible viewport, the full page, or one element, with adjustable JPEG quality. The screenshot is returned as a real image content block the model can see directly.
-Omit `target` for the whole page, or pass a nonempty Playwright selector or `aria-ref=<ref>` from a current snapshot. It must match exactly one element; select a container to read several child items. Selection does not click, scroll, or change the page. Bare ref strings such as `"e2"` are treated as selectors.
+#### Current Page Content
-Snapshots include element roles, names, refs, and viewport bounding boxes by default. Set `boxes=false` to omit boxes or use `depth` to limit the tree. Both settings are ignored for other formats.
+- **`browser_extract`**: Read the current page or one element as an AI snapshot, HTML, Markdown, or text without reloading it. Extract results after filling forms or loading more content. Snapshots include element roles, names, references, positions, and sizes for later actions. Regex search returns matching snapshot lines with nearby context and parent nodes.
-Set `search` to a nonempty Python regular expression to search snapshot lines. Matching is case-sensitive; use `(?i)` for case-insensitive matching and escape regex characters when matching them literally. Search requires snapshot output, and invalid patterns fail before capture. There is no `regex` switch. Omit `search` for the full page or target snapshot.
+#### Browser Actions
-Search runs after target selection and the depth limit. Results include all matching lines, three lines before and after each match, and parent nodes within that snapshot. Overlapping context appears once; `...` marks omitted lines. Refs and boxes are preserved. No matches returns an empty string with no status text.
+- **`browser_actions`**: Chain mouse actions, field filling, keyboard shortcuts, dialog replies, and waits in one call. Accept or dismiss browser alerts and confirmations, or enter text in prompts. For example, fill a search box, submit it, wait for results, and open a result. The AI can combine known steps while keeping snapshots separate when it needs to inspect a page change.
-HTML, Markdown, and text reuse the normal content converter. `main_content_only=true` cleans body/target content for AI; set it to `false` to skip that cleaning. This setting is ignored for snapshots. HTML is parsed output and can include document wrappers around a selected fragment. Serialized HTML does not reflect every live form property or include a host's shadow root or an iframe's document; use snapshots for live control state. Capture does not reload the page or change its forms, focus, or scroll position.
+ Reveal hover menus and tooltips, click buttons and links, or move over a nested panel and scroll it vertically or horizontally. Moves and clicks support selectors, snapshot references, or coordinates, with automatic waiting and scrolling into view for element targets. Supports coordinate movement in multiple steps, left, right, and middle clicks, plus double-clicks.
-```json
-{"session_id": "browser", "search": "(?i)next|load more"}
-```
+ Target fields anywhere on the page through selectors or snapshot references. Fill text, check or uncheck boxes, select radio buttons, and choose one or more native dropdown options by their visible labels. Text can replace existing content, clear it, or continue at the current caret. Keyboard shortcuts can select and delete text, move focus, submit forms, or dismiss menus. Slow mode types one character at a time with random pauses between characters and all actions.
-```json
-{"session_id": "browser", "extraction_type": "markdown", "target": "#results"}
-```
+ Wait for results to appear, loading overlays to disappear, or page load events before the next action. Also supports fixed pauses and waiting for network activity to settle.
-### `browser_actions` -- Chain mouse, field, keyboard, dialog, and wait actions
+#### JavaScript
-Runs actions in order on the existing page in a stealthy browser session. Call `browser_fetch` first. The flat `actions` list can mix any of the twelve action types below; use `browser_extract` when you need to inspect the page before choosing later actions.
+- **`browser_evaluate`**: Run JavaScript on the current page for custom data extraction, calculations, and page-specific tasks. The AI can collect data from several elements, read application state, or update page content. Supports asynchronous scripts and returns their results as JSON.
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID of an open browser session |
-| `actions` | list[object] | required | Nonempty ordered list; each item has a `type` and that type's fields |
-| `slowly` | bool | false | Fresh random delays of 50-150 ms per typed character and 100-300 ms between all actions, including waits |
+#### Network History
-| Action `type` | Fields | Behavior |
-|---------------|--------|----------|
-| `move` | Target, `steps=1`, `timeout=30000` | Hover over an element or move to coordinates; `steps` is the number of coordinate mousemove events, at least 1, and is ignored for element targets |
-| `click` | Target, `button="left"`, `click_count=1`, `delay=0`, `timeout=30000` | Click with the left, right, or middle button; `click_count` must be at least 1, with 2 for a double-click; `delay` is finite nonnegative milliseconds between press and release |
-| `wheel` | `delta_x=0`, `delta_y=0` | Scroll at the current pointer position; finite CSS pixel deltas allow fractions and zero; positive moves right/down, negative moves left/up |
-| `textbox` | Target, `value`, `clear=true`, `timeout=30000` | By default, replace text in an input, textarea, or contenteditable element with a string; an empty string clears it; `clear=false` types at the current caret or selection without clearing |
-| `checkbox` | Target, `value`, `timeout=30000` | Boolean `true` checks the box; `false` unchecks it |
-| `radio` | Target, `value`, `timeout=30000` | Select the radio option; `value` is required and only `true` is supported |
-| `combobox` | Target, `value`, `timeout=30000` | Select native `<select>` options by label; use a string, a list of strings for multiple selections, or `[]` to clear |
-| `press_key` | `key` | Press a nonempty native key name, character, or shortcut at the current focus |
-| `dialog` | `accept`, optional `prompt_text` | Queue one reply for the next browser dialog; `true` accepts, `false` dismisses |
-| `wait_time` | `milliseconds` | Fixed pause in finite nonnegative milliseconds |
-| `wait_element` | `target`, `state="visible"`, `timeout=30000` | Wait for an element to reach `"visible"`, `"hidden"`, `"attached"`, or `"detached"` |
-| `wait_load` | `state`, `timeout=30000` | Wait for `"domcontentloaded"`, `"load"`, or `"networkidle"` |
+- **`browser_network_requests`**: Find completed requests across tabs, page loads, and browser actions. Discover the API calls behind a page's data, inspect HTTP errors, and follow saved redirects. Static resources are hidden by default to keep results small. Returns request details as JSON, with pagination for long histories.
+- **`browser_network_request`**: Inspect one recorded request's summary, headers, sent data, or text response body as structured JSON. Returns the selected part in full without loading the page or sending the request again. Saved responses remain available after page changes. Size limits keep the history bounded; a note explains non-text, oversized, or unreadable bodies. Read needed details before closing the MCP session.
-**Targets and timing.** Element actions use a nonempty `target`: a Playwright selector such as `"#search"` or a snapshot reference as `"aria-ref=e2"`. Bare reference strings such as `"e2"` are treated as selectors. Each move or click needs either `target` or both `x` and `y`. Coordinates must be finite viewport CSS pixels from the main frame's top-left, not full-page screenshot coordinates. Field actions need `target` and can target fields anywhere on the page without a `<form>` parent. Element waits also use `target`. Mouse and field targets must match exactly one element. Use current snapshot references; do not build a chain with references that need a future snapshot.
+Response bodies are read and saved only for supported text `Content-Type` values, such as HTML, JSON, XML, or JavaScript. Binary, missing, or unrecognized content types keep request metadata and headers without reading or saving their response bodies. The browser still loads resources normally.
-Each action with a `timeout` accepts finite nonnegative milliseconds, default 30000; 0 disables the limit. Mouse coordinate actions ignore it. For text input, the limit applies separately to each native operation, including clearing and each character press in slow mode. Keyboard, dialog, wheel, and fixed-pause actions have no timeout option. Pauses between actions are outside action timeouts.
+Only completed requests are saved, including HTTP 4xx and 5xx responses. Failed and unfinished requests are omitted. Entries appear after capture finishes; reading history or closing the session does not wait. Captures still running at navigation or closure may be omitted. Saved bytes come from the Browser and can differ from the server's bytes. Recorded text follows Playwright's UTF-8 decoding when valid, with the declared charset as a fallback.
-**Mouse actions.** Element targets use native locator hover or click, which waits for readiness and scrolls into view. Coordinate actions use the native mouse without automatic waiting or scrolling. Coordinate clicks move the mouse, then press and release, without waiting for navigation. Wheel actions neither move the pointer nor wait for scrolling, animations, or loaded content to finish. The page may prevent scrolling, or the scrollable area may be at its edge.
+### Shadow DOM
-**Fields and keys.** Field actions use native locator `fill`, `press_sequentially`, `set_checked`, and `select_option(label=...)`. Combobox actions support native `<select>` elements, not custom dropdown widgets. With `clear=false`, text uses `press_sequentially` even when `slowly=false`; an empty value preserves existing content. With `slowly=true`, text clears first unless `clear=false`, then each character gets a fresh 50-150 ms delay. Every pair of actions also gets a fresh 100-300 ms pause, including explicit wait actions, with no leading or trailing pause.
+Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to include open Shadow DOM content. It defaults to `false`. See [Shadow DOM](fetching/stealthy.md#shadow-dom) for selector examples and limits.
-Each `press_key` uses native `page.keyboard.press` at the current focus. Keep a shortcut such as `"ControlOrMeta+A"` in one `key` string, then use another `press_key` action with `"Backspace"` to delete. A space (`" "`) and plus (`"+"`) are valid key strings. The native API validates key names and shortcuts when pressed. Use a click action to focus the intended control before an Enter press when needed. Key presses do not wait for navigation.
+### Key Capabilities
+- **Smart Content Extraction**: Convert web pages/elements to Markdown, HTML, or extract a clean version of the text content
+- **CSS Selector Support**: Use the Scrapling engine to target specific elements with precision before handing the content to the AI
+- **Anti-Bot Bypass**: Handle Cloudflare Turnstile, Interstitial, and other protections
+- **Proxy Support**: Use proxies for anonymity and geo-targeting
+- **Browser Impersonation**: Mimic real browsers with TLS fingerprinting, real browser headers matching that version, and more
+- **Session Persistence**: Reuse browser sessions across multiple requests for better performance
+- **Ad Blocking**: All browser-based tools automatically block requests to ~3,500 known ad and tracker domains, saving tokens and speeding up page loads
+- **Prompt Injection Protection**: Automatic sanitization of hidden content (inline hidden elements, aria-hidden, hidden slots, zero-width characters, HTML comments, template tags) that could be used for prompt injection attacks
-**Dialogs.** Place `dialog` before the click, key, or other action that opens an alert, confirmation, prompt, or leave-page warning (`beforeunload`). Each reply handles one dialog; queue several replies before a trigger that opens several dialogs. Replies are used in order. For prompts, `prompt_text` supplies the accepted text; omitting it or passing an empty string returns empty text, even if the prompt has a default. It has no effect when dismissing or accepting other dialog types. Unused replies expire when the call ends; the action does not wait for a dialog by itself. For delayed dialogs, include a wait for the resulting page change in the same chain. HTML modal windows use normal element actions.
+#### But why use Scrapling MCP Server instead of other available tools?
-```json
-{"session_id": "browser", "actions": [{"type": "dialog", "accept": true, "prompt_text": "books"}, {"type": "click", "target": "#open-prompt"}]}
-```
+Aside from its stealth capabilities and ability to bypass Cloudflare Turnstile/Interstitial, Scrapling's server is the only one that lets you select specific elements to pass to the AI, saving a lot of time and tokens!
-**Waits.** Element waits are strict: multiple matches return an error. `attached` means present in the DOM, and `detached` means absent. `visible` requires a nonempty bounding box and no `visibility:hidden`; `hidden` also succeeds when the element is absent. Load waits observe the current committed document and return immediately if the state was already reached. A click or key press followed by `wait_load` does not guarantee waiting for delayed future navigation. `domcontentloaded` waits for DOMContentLoaded, not all future JavaScript work; `load` waits for the load event; `networkidle` requires no active network connections for at least 500 ms and does not prove application readiness. Prefer a specific result element when it signals readiness.
+The way other servers work is that they extract the content, then pass it all to the AI to extract the fields you want. This causes the AI to consume far more tokens than needed (from irrelevant content). Scrapling solves this problem by allowing you to pass a CSS selector to narrow down the content you want before passing it to the AI, which makes the whole process much faster and more efficient.
-For example, fill and submit a search, wait for its results, then hover over the results panel and scroll it:
+## Installation
-```json
-{
- "session_id": "browser",
- "actions": [
- {"type": "textbox", "target": "#search", "value": "books"},
- {"type": "press_key", "key": "Enter"},
- {"type": "wait_element", "target": "#results"},
- {"type": "move", "target": "#results"},
- {"type": "wheel", "delta_y": 500}
- ]
-}
-```
+Install Scrapling with MCP Support, then double-check that the browser dependencies are installed.
-MCP validates the entire action schema, and all target combinations are checked before execution. Targets are resolved only when their action runs, so earlier actions can create later targets. One page stays reserved for the whole sequence, including waits and random pauses. The first runtime failure stops the chain and reports its one-based action number, type, and native error. Earlier effects remain, and the failed action itself may partly apply. Nothing is retried or rolled back. Cancellation also stops the chain and remains cancellation. A cancelled or timed-out click attempts to release its button while the page is open, preserving the original error if release fails. The page reservation is released after success, failure, or cancellation.
+```bash
+# Install Scrapling with MCP server dependencies
+pip install "scrapling[ai]"
-Returns `Actions completed.` as plain text without echoing field values or taking a snapshot. Use `browser_extract` before retrying after an error or when later actions depend on inspecting a page change. Unknown or HTTP sessions and missing, closed, or busy pages return an error.
+# Install browser dependencies
+scrapling install
+```
-### `browser_evaluate` -- Run JavaScript on the current page
+Or use the Docker image directly from the Docker registry:
+```bash
+docker pull pyd4vinci/scrapling
+```
+Or download it from the GitHub registry:
+```bash
+docker pull ghcr.io/d4vinci/scrapling:latest
+```
-Runs a native JavaScript expression or function on the existing page. Call `browser_open`, then `browser_fetch` first. Functions are invoked with the optional JSON object argument, and returned promises are awaited.
+## Setting up the MCP Server
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID of an open browser session with a page |
-| `expression` | str | required | Nonempty JavaScript expression or function to invoke |
-| `arg` | object or null | null | One JSON object passed to the function; nested values can contain any JSON data |
-| `isolated_context` | bool | true | Use a separate JavaScript context; `false` accesses the page's own variables |
+The setup below works with [Claude Desktop](https://claude.ai/download), [Claude Code](https://www.anthropic.com/claude-code), and other clients that support MCP.
-The isolated context shares the page's DOM but does not share its JavaScript variables. Use `isolated_context=false` to access application globals.
+**Note:**
+The `scrapling-mcp` command used below was added in v0.4.13 as a shortcut that maps directly to `scrapling mcp`, making it easier to add Scrapling to MCP registries and clients that expect a single command. If you are on an older version, use the `scrapling` command with `mcp` as the first argument instead.
-For example, collect text from the first matching links:
+### Claude Desktop
+Scrapling MCP server configuration:
+```json
+"ScraplingServer": {
+ "command": "scrapling-mcp"
+}
+```
+If that's the first MCP server you're adding, set the content of the file to this:
```json
{
- "session_id": "browser",
- "expression": "({selector, limit}) => [...document.querySelectorAll(selector)].slice(0, limit).map(element => element.textContent)",
- "arg": {"selector": "a", "limit": 10}
+ "mcpServers": {
+ "ScraplingServer": {
+ "command": "scrapling-mcp"
+ }
+ }
}
```
+Configuration file locations:
-Return JSON-compatible data: null, strings, booleans, finite numbers, arrays, or objects. Convert special objects such as DOM nodes, dates, maps, and sets into plain data in JavaScript. Both `null` and `undefined` return `null`. Circular data, non-finite numbers, and other results that cannot be encoded as JSON return an error.
-
-The result is compact JSON in one `TextContent` block with `structured_output=False`: strings are JSON-quoted, arrays stay in one block, and null is retained. No automatic snapshot, navigation, reload, retry, or tool timeout is added.
-
-The page stays reserved during evaluation and is released after success, failure, or cancellation. Scripts can change page state or make requests. Errors and cancellation do not undo those effects or guarantee that browser-side JavaScript stops. Use `browser_extract` or another read to check effects before repeating a script. Unknown or HTTP sessions and missing, closed, or busy pages return an error; JavaScript errors propagate.
-
-### `browser_network_requests` -- Search recorded browser requests
-
-Reads completed requests from a session opened with `browser_open`, across page loads, actions, tabs, and navigation. It does not navigate, reserve a page, or send requests. Recording starts automatically and saves response headers and supported text bodies. Completed HTTP 4xx and 5xx responses are included; failed and unfinished requests are omitted. The history retains up to 1,000 requests, with fixed limits of 1 MiB per response body and 20 MiB combined; old records are removed when retention limits are reached.
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID from `browser_open` |
-| `url_pattern` | str or null | null | Regular expression to match request URLs |
-| `include_static` | bool | false | Include all resource types; otherwise keep XHR/fetch and HTTP errors |
-| `after_id` | int | 0 | Nonnegative cursor; return requests with larger IDs |
-| `limit` | int | 50 | Maximum returned requests, from 1 to 200 |
-
-Returns structured JSON with `requests`, `next_cursor`, `has_more`, and `dropped_count`. Each request contains integer `id` and `status` fields, plus string `url`, `method`, and `resource_type` fields. `dropped_count` counts older entries removed by the history limits.
-
-Pass `next_cursor` as `after_id` to read later entries. `has_more` means more retained entries currently match the same filters. An empty result has `requests=[]` and `has_more=false`; `next_cursor` is the greater of `after_id` and the latest saved request ID. Use `include_static=true` to include successful document requests and saved redirects. Entries receive increasing IDs when capture finishes. Reads do not wait for captures; use browser actions to wait for the page when needed.
-
-Both network tools provide an MCP output schema and `structured_content`. The SDK also returns the JSON in a text block for clients that read text content.
-
-### `browser_network_request` -- Inspect one recorded request
-
-Reads one recorded request without navigating, reserving a page, or sending it again. All parts use saved data, which remains available after page changes and temporary-context closure. Reads do not call browser methods. An entry appears only after its response has been converted and saved.
-
-The recorder reads and saves response bodies only for supported text `Content-Type` values: `text/*`, `+json`/`+xml` types (including SVG), JSON, XML, JavaScript, GraphQL, and URL-encoded form data. Binary or unrecognized types, and responses without `Content-Type`, keep their metadata but no body bytes. The browser still loads resources normally.
-
-Saved response bytes come from Playwright and can differ from the server's original bytes. Recorded text uses UTF-8 when the saved bytes are valid UTF-8, matching Playwright; otherwise it uses the declared charset. Response headers stay unchanged.
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID from `browser_open` |
-| `request_id` | int | required | Positive recorded request ID from `browser_network_requests` |
-| `part` | str | `"summary"` | `"summary"`, `"request_headers"`, `"request_body"`, `"response_headers"`, or `"response_body"` |
-
-Returns structured JSON with `request_id`, `part`, `data`, and `note`.
+1. **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
+2. **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
-- Summary `data` is an object with `id`, `url`, `method`, `resource_type`, and `status`.
-- Header `data` is an object of header names and values. Partial headers keep that object and add a `note`.
-- Body `data` is the full saved text for any text resource type, including documents and API calls. JSON response bodies stay strings; they are not parsed into objects.
+To ensure it's working, use the full path to the `scrapling-mcp` executable:
-Saved empty text, HEAD responses, and status codes 204, 205, and 304 return `data=""` and `note=null`. Missing, unreadable, oversized, or non-text body data returns `data=null` and a `note` with the reason. Skipped non-text responses use `Non-text body; not saved.`. A missing or evicted request ID returns an error.
-
-Saved response data survives browser cleanup. Closing a session does not wait for captures; captures still running at navigation or closure may be omitted. The limits cover retained request entries and response bytes, not total browser memory, concurrent captures, temporary full-body reads, parsed HTML, or request bodies.
-
-Both network tools reject unknown or HTTP sessions. Read needed details before `close_session`, which removes the MCP session.
-
-### `session_make_request` -- HTTP request through an open requests session
-
-Makes an HTTP request (any method) through a session opened with `open_request_session`, reusing its cookies, connections, and browser fingerprint across calls. Same parameters as `make_request` plus a required `session_id`, minus the session-level `impersonate`, `proxy`, and `proxy_auth`. Raises on a browser session.
-
-### `close_session` -- Close a persistent session
-
-Closes a session (browser or requests) and frees its resources. Always close sessions when done.
-
-| Parameter | Type | Default | Description |
-|--------------|------|----------|----------------------------------|
-| `session_id` | str | required | Session ID from `browser_open` |
-
-Returns a `SessionClosedModel` with `session_id` and `message`.
-
-### `list_sessions` -- List active sessions
-
-Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, `is_alive`, and `settings` (same as `browser_open` returns).
-
-No parameters.
-
-### `browser_screenshot` -- Capture the current page or an element
-
-Captures the existing page or one element without reloading or navigating. Element capture can scroll the target into view. Returns an MCP `ImageContent` block followed by a `TextContent` block with the current page URL. The tool uses `structured_output=False`, so the model receives real image content rather than image data duplicated in a structured JSON result.
-
-Call `browser_open`, then `browser_fetch` to open a page first. Use `browser_actions` for waits or interactions before capture; this tool does not accept a URL or readiness controls.
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `session_id` | str | required | ID of an open browser session with a page |
-| `selector` | str or null | null | Nonempty Playwright selector for one element; cannot combine with `ref` |
-| `ref` | str or null | null | Nonempty current snapshot reference, such as `e2`; cannot combine with `selector` |
-| `image_type` | `"png"` / `"jpeg"` | `"png"` | Image format; JPEG allows smaller payloads |
-| `full_page` | bool | false | Capture the full scrollable page; cannot combine with an element target |
-| `quality` | int or null | null | JPEG quality 0-100; an error if supplied for PNG |
-| `timeout` | number | 30000 | Finite nonnegative capture timeout in milliseconds; 0 disables it |
-
-Omit both `selector` and `ref` for the viewport or full-page capture. For an element, pass exactly one target and leave `full_page=false`. Native element capture requires exactly one match, waits for the element to be ready, and may scroll it into view. Refresh stale references with `browser_extract`.
-
-The element image is clipped to its bounding box. A scrollable container includes only its currently visible contents, not all content inside it. Content covered by another element stays covered in the image.
-
-For example, capture a result card through its selector:
+1. **MacOS**: `which scrapling-mcp`
+2. **Windows**: `where scrapling-mcp`
+For me, on my Mac, it returned `/Users/<MyUsername>/.venv/bin/scrapling-mcp`, so the config I used in the end is:
```json
-{"session_id": "browser", "selector": "#result-card"}
+{
+ "mcpServers": {
+ "ScraplingServer": {
+ "command": "/Users/<MyUsername>/.venv/bin/scrapling-mcp"
+ }
+ }
+}
```
-
-Or use a reference from the current snapshot:
-
+#### Docker
+If you are using the Docker image, then it would be something like
```json
-{"session_id": "browser", "ref": "e2"}
-```
-
-The page stays reserved during capture and is released after success, failure, or cancellation. Unknown or HTTP sessions and missing, closed, or busy pages return an error. Capture errors propagate.
-
-## Tool selection guide
-
-| Scenario | Tool |
-|------------------------------------------|---------------------------------------------------------------|
-| Single static page or API request | `make_request` |
-| Single JavaScript-rendered / SPA page | `browser_fetch_once` |
-| Single page with Cloudflare protection | `browser_fetch_once` with `solve_cloudflare=true` |
-| Related HTTP requests | `open_request_session` + `session_make_request` per URL |
-| Reuse browser login state across sessions | `browser_save_state` / `browser_load_state` with `session_id` and `path` |
-| Browser automation or network history | `browser_open` + `browser_fetch` per URL |
-| Capture the current page or an element | `browser_screenshot` after `browser_open` + `browser_fetch` |
-| Read the current page or one element as a snapshot, HTML, Markdown, or text | `browser_extract` with `session_id` |
-| Chain mouse, field, keyboard, or wait actions | `browser_actions` with `session_id` |
-| Custom JavaScript extraction or page tasks | `browser_evaluate` with `session_id` |
-| Find API calls, HTTP errors, or redirects | `browser_network_requests` with `session_id` |
-| Read recorded request headers or body data | `browser_network_request` with `session_id` and `request_id` |
-
-Start with `make_request` for a standalone HTTP request, then use `browser_fetch_once` if JavaScript rendering is needed or HTTP requests are blocked. Enable `solve_cloudflare` for Cloudflare challenges. Use persistent sessions for related requests, browser actions, screenshots, or network history. Reuse the session and fetch URLs in order. Always call `close_session` when done with a persistent session.
-
-## Content extraction tips
-
-- Use `css_selector` to narrow results before they reach the model -- this saves significant tokens.
-- `main_content_only=true` (default) restricts HTML, Markdown, and text extraction to `<body>`.
-- `extraction_type="markdown"` (default) is best for readability. Use `"text"` for minimal output, `"html"` when structure matters.
-- If a `css_selector` matches multiple elements, HTML, Markdown, and text extraction return all matches in the `content` list. Snapshot extraction requires exactly one match.
-
-## Prompt injection protection
-
-For HTML, Markdown, and text extraction, `main_content_only=true` (the default) sanitizes scraped content to prevent prompt injection from malicious websites. It strips:
-
-- CSS-hidden elements (`display:none`, `visibility:hidden`, `opacity:0`, `font-size:0`, `height:0`, `width:0`)
-- `aria-hidden="true"` elements
-- `<template>` tags
-- HTML comments
-- Zero-width unicode characters
-
-Keep `main_content_only=true` for maximum protection.
-
-## Ad blocking
-
-The `browser_fetch_once` tool and browser sessions opened with `browser_open` automatically block requests to ~3,500 known ad and tracker domains. This is always enabled in the MCP server to save tokens and speed up page loads. No configuration needed.
-
-## Setup
-
-Start the server (stdio transport, used by most MCP clients):
-
-```bash
-scrapling-mcp
+{
+ "mcpServers": {
+ "ScraplingServer": {
+ "command": "docker",
+ "args": [
+ "run", "-i", "--rm", "pyd4vinci/scrapling", "mcp"
+ ]
+ }
+ }
+}
```
-**Note:** The `scrapling-mcp` command was added in v0.4.13 as a shortcut that maps directly to `scrapling mcp`, making it easier to add Scrapling to MCP registries and clients that expect a single command. On older versions, use the `scrapling` command with `mcp` as the first argument instead.
+The same logic applies to [Cursor](https://cursor.com/docs/context/mcp), [WindSurf](https://windsurf.com/university/tutorials/configuring-first-mcp-server), and others.
-Or with Streamable HTTP transport:
+### Claude Code
+Here it's much simpler to do. With [Claude Code](https://www.anthropic.com/claude-code):
```bash
-scrapling-mcp --http
-scrapling-mcp --http --host 0.0.0.0 --port 8000
+claude mcp add ScraplingServer "/Users/<MyUsername>/.venv/bin/scrapling-mcp"
```
+To get Scrapling's executable path:
-The host defaults to `127.0.0.1`, so the server only accepts connections from the same machine. Pass `--host 0.0.0.0` to make it reachable from the network.
-
-Docker alternative:
+1. **MacOS**: `which scrapling-mcp`
+2. **Windows**: `where scrapling-mcp`
-```bash
-docker pull pyd4vinci/scrapling
-docker run -i --rm pyd4vinci/scrapling mcp
-```
+Here's the main article from Anthropic on [how to add MCP servers to Claude code](https://docs.anthropic.com/en/docs/claude-code/mcp#option-1%3A-add-a-local-stdio-server) for further details.
-That runs the stdio transport. To use Streamable HTTP inside Docker, bind to `0.0.0.0` yourself and set a token, since the container's `127.0.0.1` is not reachable through the published port:
-```bash
-docker run -p 8000:8000 -e SCRAPLING_MCP_AUTH_TOKEN="<your-token>" pyd4vinci/scrapling mcp --http --host 0.0.0.0
-```
+Restart the client after adding the server.
-## Custom browser executable
+### Custom Browser Executable
The `browser_fetch_once` and `browser_open` tools can use a custom Chromium-compatible browser executable instead of the bundled Chromium. This is useful for custom browser builds or lightweight browser engines.
@@ -465,31 +209,56 @@ In a Claude Desktop configuration, add the option to the server arguments:
}
```
-You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. Pass `executable_path` to `browser_fetch_once` or `browser_open` when a single fetch or session needs a different browser executable. The `scrapling extract fetch` CLI command supports the same `--executable-path` option and environment variable fallback.
-
-The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling-mcp` binary with no arguments (or the `scrapling` binary with `mcp` as the argument on versions before 0.4.13).
+You can also set the `SCRAPLING_EXECUTABLE_PATH` environment variable before starting the server. The AI can still choose a different browser executable for a single fetch with `browser_fetch_once` or when opening a session with `browser_open`.
-## Connecting to remote browsers
+### Connecting to Remote Browsers
-`browser_open` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, `browser_network_request`, `browser_save_state`, and `browser_load_state` as usual.
+`browser_open` doesn't have to launch a browser locally. Pass a CDP url and it will connect to an already-running browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/), whether that browser is on the same machine, another host, or a managed browser provider:
+```
+Open a stealthy browser session on wss://cdp.provider.example/session/abc123, then use it to scrape the product details from https://shop.example.com. Close the session when you're done.
+```
+The `session_id` you get back is used with `browser_fetch`, `browser_extract`, `browser_actions`, `browser_evaluate`, `browser_screenshot`, `browser_network_requests`, `browser_network_request`, `browser_save_state`, and `browser_load_state` as usual.
-The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.
+The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser you started yourself with the remote debugging port enabled:
+```commandline
+chrome --remote-debugging-port=9222
+```
+That one is reached with `cdp_url="http://localhost:9222"`, or with the host's address if the browser is running on another machine.
**Notes:**
-- The browser is already running, so options that only apply while launching one are ignored for CDP sessions: `headless`, `real_chrome`, and `executable_path` (including the server-wide default).
-- Everything else still applies (`locale`, `useragent`, `proxy`, `cookies`, `timezone_id`, and so on), as each session creates its own browser context on the remote browser.
+* The browser is already running, so options that only apply while launching one are ignored for CDP sessions: `headless`, `real_chrome`, and `executable_path` (including the server-wide default above).
+* Everything else still applies (`locale`, `useragent`, `proxy`, `cookies`, `timezone_id`, and so on), as each session creates its own browser context on the remote browser.
-## Authentication
+### Streamable HTTP
+Since version 0.3.6, we have added the ability to make the MCP server use the 'Streamable HTTP' transport mode instead of the traditional 'stdio' transport.
-The stdio transport is only reachable by the program that started it, but with Streamable HTTP anyone who can reach the port can call every tool, including fetching any URL from the machine running the server. That's why Streamable HTTP requires authentication, so `--http` on its own refuses to start and asks you for a token:
+So instead of using the following command (the 'stdio' one):
+```bash
+scrapling-mcp
+```
+Use the following to enable 'Streamable HTTP' transport mode:
+```bash
+scrapling-mcp --http
+```
+Hence, the default value for the host the server is listening to is '127.0.0.1' and the port is 8000, which both can be configured as below:
+```bash
+scrapling-mcp --http --host '0.0.0.0' --port 8000
+```
+The default only accepts connections from the same machine. Pass `--host '0.0.0.0'` when you want the server to be reachable from the network, which is a separate decision from authentication below.
+If you run the 'Streamable HTTP' transport inside Docker, you have to bind to '0.0.0.0' yourself and set a token, otherwise the published port can't reach the server (the container's '127.0.0.1' is only visible inside the container):
```bash
-scrapling-mcp --http --auth-token "$(openssl rand -hex 32)"
+docker run -p 8000:8000 -e SCRAPLING_MCP_AUTH_TOKEN="<your-token>" pyd4vinci/scrapling mcp --http --host '0.0.0.0'
```
-Clients then have to send that token in an `Authorization` header, and any request without it is rejected with a `401`:
+### Authentication
+The 'stdio' transport is only reachable by the program that started it, but the moment you switch to 'Streamable HTTP', anyone who can reach the port can call every tool, and that includes fetching any URL from the machine running the server. That's why 'Streamable HTTP' requires authentication, so `--http` on its own refuses to start and asks you for a token:
+```bash
+scrapling-mcp --http --auth-token "$(openssl rand -hex 32)"
+```
+Clients then have to send that token in an `Authorization` header, and any request without it is rejected with a `401`:
```json
{
"mcpServers": {
@@ -502,32 +271,190 @@ Clients then have to send that token in an `Authorization` header, and any reque
}
}
```
-
-Passing the token on the command line leaves it in the shell history and the process list, so prefer the `SCRAPLING_MCP_AUTH_TOKEN` environment variable:
-
+Passing the token on the command line leaves it in your shell history and in the process list, so prefer the `SCRAPLING_MCP_AUTH_TOKEN` environment variable:
```bash
export SCRAPLING_MCP_AUTH_TOKEN="<your-token>"
scrapling-mcp --http
```
-
-If you really want an unauthenticated server, for example while testing locally on the default `127.0.0.1`, you have to ask for it with `--no-auth`:
-
+If you really want an unauthenticated server, for example while testing locally on the default '127.0.0.1', you have to ask for it with `--no-auth`:
```bash
scrapling-mcp --http --no-auth
```
+Combining `--no-auth` with `--host '0.0.0.0'` leaves every tool open to anyone who can reach the port, so avoid that pair outside a trusted network.
-Combining `--no-auth` with `--host 0.0.0.0` leaves every tool open to anyone who can reach the port, so avoid that pair outside a trusted network.
-
-When the server listens on a public address, also tell it which host names to accept, which turns on protection against DNS-rebinding attacks. The option can be repeated:
-
+When the server listens on a public address, you should also tell it which host names to accept, which turns on protection against DNS-rebinding attacks (a website your browser visits trying to talk to your server). The option can be repeated:
```bash
scrapling-mcp --http --allowed-host 'your-server.example.com:8000'
```
**Notes:**
-- Authentication applies to the Streamable HTTP transport only. It's ignored with stdio, and the server logs a warning to say so.
-- Plain HTTP sends the token in cleartext, so put the server behind a reverse proxy that terminates TLS before exposing it to the internet.
-- This is a single shared key, not per-client credentials, so every client uses the same token and rotating it means restarting the server.
-- Starting with `--http --no-auth` still logs a warning that it's unauthenticated.
-- Passing both `--auth-token` and `--no-auth` keeps the token, so the server stays authenticated instead of quietly dropping it.
+* Authentication applies to the 'Streamable HTTP' transport only. It's ignored with 'stdio', and the server logs a warning to tell you so.
+* Plain HTTP sends the token in cleartext, so put the server behind a reverse proxy that terminates TLS before exposing it to the internet.
+* This is a single shared key, not per-client credentials, so every client uses the same token, and rotating it means restarting the server.
+* Starting the server with `--http --no-auth` still logs a warning telling you that it's unauthenticated.
+* Passing both `--auth-token` and `--no-auth` keeps the token, so the server stays authenticated instead of quietly dropping it.
+
+## Examples
+
+1. **Basic Web Scraping**
+
+ Extract the main content from a webpage as Markdown:
+
+ ```
+ Scrape the main content from https://example.com and convert it to markdown format.
+ ```
+
+ Claude can use `make_request` to fetch the page and return clean, readable content. HTTP requests default to three attempts with a one-second delay between attempts. If the page needs JavaScript or blocks HTTP requests, the AI can use `browser_fetch_once` instead. Both tools close their own resources after the call.
+
+ A more optimized version of the same prompt would be:
+ ```
+ Use regular requests to scrape the main content from https://example.com and convert it to markdown format.
+ ```
+
+2. **Targeted Data Extraction**
+
+ Extract specific elements using CSS selectors:
+
+ ```
+ Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds.
+ ```
+
+ The server will extract only the elements matching your selector and return them as a structured list. The prompt sets five attempts for connection issues.
+
+3. **E-commerce Data Collection**
+
+ Another example of a bit more complex prompt:
+ ```
+ Open a stealthy browser session and visit these e-commerce URLs one at a time:
+ - https://shop1.com/product-a
+ - https://shop2.com/product-b
+ - https://shop3.com/product-c
+
+ Get the product names, prices, and descriptions from each page. Close the session when done.
+ ```
+
+ Claude can reuse one browser session, call `browser_fetch` for each URL in order, then analyze the extracted data.
+
+4. **More advanced workflow**
+
+ Get all the action games on PlayStation's store first page:
+ ```
+ Extract the URLs of all games on this page, then visit each game in order and return a list of all action games: https://store.playstation.com/en-us/pages/browse
+ ```
+ Reuse an HTTP session to avoid launching a browser when the data is available through normal requests:
+ ```
+ Open an HTTP session, extract the URLs of all games on this page, then fetch each game in order and return a list of all action games. Close the session when done: https://store.playstation.com/en-us/pages/browse
+ ```
+ Use CSS selectors to narrow the content before the AI reads it:
+ ```
+ Use an HTTP session to extract the URLs of all games on the page below, then fetch each game in order and return a list of all action games.
+ The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in each game page is `[data-qa="gameInfo#releaseInformation#genre-value"]`.
+ Close the session when done.
+
+ URL: https://store.playstation.com/en-us/pages/browse
+ ```
+
+5. **Get data from a website with Cloudflare protection**
+
+ ```
+ What's the price of this product? Be cautious, as it utilizes Cloudflare's Turnstile protection. Make the browser visible while you work.
+
+ https://ao.com/product/oo101uk-ninja-woodfire-outdoor-pizza-oven-brown-99357-685.aspx
+ ```
+
+6. **Long workflow**
+
+ You can, for example, use a prompt like this:
+ ```
+ Extract all product URLs for the following category, then return the prices and details for the first 3 products.
+
+ https://www.arnotts.ie/furniture/bedroom/bed-frames/
+ ```
+ But a better prompt would be:
+ ```
+ Go to the following category URL and extract all product URLs using the CSS selector "a". Then, reuse the session to fetch the first 3 product pages in order and extract each product’s price and details. Close the session when done.
+
+ Keep the output in markdown format to reduce irrelevant content.
+
+ Category URL:
+ https://www.arnotts.ie/furniture/bedroom/bed-frames/
+ ```
+
+7. **Using Persistent Sessions**
+
+ When scraping multiple pages from the same site, use a persistent browser session to avoid the overhead of launching a new browser for each request:
+ ```
+ Open a stealthy browser session, then use it to scrape the main details from the first 5 product pages on https://shop.example.com. Close the session when you're done.
+ ```
+ Claude will use `browser_open` to create a persistent browser, call `browser_fetch` for each product page through that session, and then call `close_session` at the end. This is significantly faster than launching a new browser for each page.
+
+ **Danger:**
+
+ When using persistent sessions, always remember to close the session after you finish or it will stay open!
+
+
+8. **Using Persistent Session on a long flow**
+
+ ```
+ Use Scrapling MCP to do the following in this order:
+
+ 1. Open a stealthy browser session with headless mode off.
+ 2. Go to this page and collect the number of stars: https://github.com/D4Vinci/Scrapling
+ 3. From the README, get the URL that shows the number of downloads and go to it.
+ 4. Get the number of downloads and the top 3 countries from the graph.
+ 5. Prepare a report with the results.
+ 6. Close the browser.
+ ```
+
+## Best Practices
+
+### 1. Choose the Right Tool
+- **`make_request`**: Fast HTTP requests for a single page or API call
+- **`browser_fetch_once`**: A single JavaScript/dynamic page or protected site, including Cloudflare
+- **Session tools**: Related requests, browser actions, screenshots, and network history
+
+### 2. Optimize Performance
+- Reuse a session and fetch each URL in order
+- Disable unnecessary resources
+- Set appropriate timeouts
+- Use CSS selectors for targeted extraction
+
+### 3. Handle Dynamic Content
+- Use `network_idle` for SPAs
+- Set `wait_selector` for specific elements
+- Increase timeout for slow-loading sites
+
+### 4. Data Quality
+- Use `main_content_only=true` to select and clean the page's body
+- Choose an appropriate `extraction_type` for your use case
+
+### 5. Prompt Injection Protection
+The MCP server automatically sanitizes scraped content when `main_content_only` is enabled (the default). This strips hidden content that malicious websites could use to inject instructions into the AI's context:
+
+- **Inline hidden styles**: `display:none`, `visibility:hidden`, and zero values for opacity, font size, height, width, max-height, and max-width
+- **Accessibility-hidden elements**: `aria-hidden="true"`
+- **Template tags and hidden slots**: `<template>` and `<slot hidden>` elements
+- **HTML comments**: `<!-- ... -->`
+- **Zero-width characters**: Invisible unicode characters like zero-width spaces
+
+This protection runs automatically for HTML, Markdown, and text extraction. Keep `main_content_only=true` (the default) for maximum protection.
+
+### 6. Manage Sessions
+- Use `make_request` or `browser_fetch_once` for a standalone fetch; they close their own resources and do not leave a session for later calls
+- Use `browser_open` to create a persistent browser session when scraping multiple pages, then call `browser_fetch` for each page through that session
+- For multiple plain HTTP requests, use `open_request_session` instead and call `session_make_request` per request; it keeps cookies, connections, and the browser fingerprint (`impersonate`) across calls without a browser
+- Sessions hold the session-level configuration set when opened (headless, locale, cookies, stealth toggles, etc. for browsers; `impersonate` and `proxy` for requests sessions); the per-request options (timeout, wait_selector, network_idle, solve_cloudflare, etc.) are passed to `browser_fetch`/`session_make_request` on each call, with their defaults shown in the tool schemas
+- `browser_fetch` can solve Cloudflare challenges through the open stealthy browser session
+- Always close sessions with `close_session` when done to free resources
+- Use `list_sessions` to check which sessions are still active and see the `settings` each was created with (returned for the AI agent; empty for CDP sessions)
+- Pass a custom `session_id` to the open tools to give sessions meaningful names (e.g. `"search"`, `"checkout"`) instead of the random hex default. They raise if the chosen ID is already in use, so you can detect collisions up front
+
+### 7. Capturing Screenshots
+
+- `browser_screenshot` captures the page already opened through `browser_fetch` in a stealthy browser session. It does not reload the page, so the AI can inspect filled fields, open menus, and other results of its actions.
+- The image is returned as a real `ImageContent` block, so the model sees the page directly.
+- Capture the visible viewport or the full page, including content below the fold.
+- Capture one field, chart, card, or result to inspect it with less surrounding content. Element capture can scroll the target into view.
+- Choose PNG for lossless images or JPEG with adjustable quality for smaller payloads.
+- The AI can use `browser_actions` to wait for content or finish interactions before capture.
diff --git a/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md b/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md
index 367c028..1a929f6 100644
--- a/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md
+++ b/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md
@@ -1,8 +1,10 @@
# Migrating from BeautifulSoup to Scrapling
-API comparison between BeautifulSoup and Scrapling. Scrapling is faster, provides equivalent parsing capabilities, and adds features for fetching and handling modern web pages.
+If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
-Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoids those shortcuts to preserve performance.
+Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
+
+You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code |
@@ -12,7 +14,7 @@ Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoi
| Finding a single element | `element = soup.find('div', class_='example')` | `element = page.find('div', class_='example')` |
| Finding multiple elements | `elements = soup.find_all('div', class_='example')` | `elements = page.find_all('div', class_='example')` |
| Finding a single element (Example 2) | `element = soup.find('div', attrs={"class": "example"})` | `element = page.find('div', {"class": "example"})` |
-| Finding a single element (Example 3) | `element = soup.find(re.compile("^b"))` | `element = page.find(re.compile("^b"))`<br/>`element = page.find_by_regex(r"^b")` |
+| Finding a single element (Example 3) | `element = soup.find(re.compile("^b"))` | `element = page.find(lambda e: re.search(r"^b", e.tag) is not None)` |
| Finding a single element (Example 4) | `element = soup.find(lambda e: len(list(e.children)) > 0)` | `element = page.find(lambda e: len(e.children) > 0)` |
| Finding a single element (Example 5) | `element = soup.find(["a", "b"])` | `element = page.find(["a", "b"])` |
| Find element by its text content | `element = soup.find(text="some text")` | `element = page.find_by_text("some text", partial=False)` |
@@ -30,8 +32,8 @@ Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoi
| Searching for an element in the parents of an element | `target_parent = element.find_parent("a")` | `target_parent = element.find_ancestor(lambda p: p.tag == 'a')` |
| Get all siblings of an element | N/A | `siblings = element.siblings` |
| Get next sibling of an element | `next_element = element.next_sibling` | `next_element = element.next` |
-| Searching for an element in the siblings of an element | `target_sibling = element.find_next_sibling("a")`<br/>`target_sibling = element.find_previous_sibling("a")` | `target_sibling = element.siblings.search(lambda s: s.tag == 'a')` |
-| Searching for elements in the siblings of an element | `target_sibling = element.find_next_siblings("a")`<br/>`target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')` |
+| Searching for an element in the siblings of an element | `target_sibling = element.find_next_sibling("a")` `target_sibling = element.find_previous_sibling("a")` | `target_sibling = element.siblings.search(lambda s: s.tag == 'a')` |
+| Searching for elements in the siblings of an element | `target_sibling = element.find_next_siblings("a")` `target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')` |
| Searching for an element in the next elements of an element | `target_parent = element.find_next("a")` | `target_parent = element.below_elements.search(lambda p: p.tag == 'a')` |
| Searching for elements in the next elements of an element | `target_parent = element.find_all_next("a")` | `target_parent = element.below_elements.filter(lambda p: p.tag == 'a')` |
| Searching for an element in the ancestors of an element | `target_parent = element.find_previous("a")` ¹ | `target_parent = element.path.search(lambda p: p.tag == 'a')` |
@@ -39,14 +41,17 @@ Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoi
| Get previous sibling of an element | `prev_element = element.previous_sibling` | `prev_element = element.previous` |
| Navigating to children | `children = list(element.children)` | `children = element.children` |
| Get all descendants of an element | `children = list(element.descendants)` | `children = element.below_elements` |
-| Filtering a group of elements that satisfies a condition | `group = soup.find('p', 'story').css.filter('a')` | `group = page.find_all('p', 'story').filter(lambda p: p.tag == 'a')` |
+| Select anchors inside story paragraphs | `group = soup.select('p.story a')` | `group = page.css('p.story a')` |
+A compiled regex passed directly to Scrapling's `find()` or `find_all()` matches element text, not tag names. Use the lambda shown above for tag names, or `page.find_by_regex(r"^b")` for direct text.
¹ **Note:** BS4's `find_previous`/`find_all_previous` searches all preceding elements in document order, while Scrapling's `path` only returns ancestors (the parent chain). These are not exact equivalents, but ancestor search covers the most common use case.
-BeautifulSoup supports modifying/manipulating the parsed DOM. Scrapling does not - it is read-only and optimized for extraction.
+**One key point to remember**: BeautifulSoup offers features for modifying and manipulating the page after it has been parsed. Scrapling focuses more on scraping the page faster for you, and then you can do what you want with the extracted information. So, two different tools can be used in Web Scraping, but one of them specializes in Web Scraping :)
+
+### Putting It All Together
-### Full Example: Extracting Links
+Here's a simple example of scraping a web page to extract all the links using BeautifulSoup and Scrapling.
**With BeautifulSoup:**
@@ -71,16 +76,16 @@ from scrapling import Fetcher
url = 'https://example.com'
page = Fetcher.get(url)
-links = page.css('a::attr(href)')
+links = page.css('a::attr(href)').getall()
for link in links:
print(link)
```
-Scrapling combines fetching and parsing into a single step.
+As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
-**Note:**
+**Additional Notes:**
-- **Parsers**: BeautifulSoup supports multiple parser engines. Scrapling always uses `lxml` for performance.
-- **Element Types**: BeautifulSoup elements are `Tag` objects; Scrapling elements are `Selector` objects. Both provide similar navigation and extraction methods.
-- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.find()`). `page.css()` returns an empty `Selectors` list when no elements match. Use `page.css('.foo').first` to safely get the first match or `None`.
-- **Text Extraction**: Scrapling's `TextHandler` provides additional text processing methods such as `clean()` for removing extra whitespace, consecutive spaces, or unwanted characters.
\ No newline at end of file
+- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
+- **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
+- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.find()`). In Scrapling, `page.css()` returns an empty `Selectors` list when no elements match, and you can use `page.css('.foo').first` to safely get the first match or `None`. To avoid errors, check for `None` or empty results before accessing properties.
+- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters.
diff --git a/agent-skill/Scrapling-Skill/references/parsing/adaptive.md b/agent-skill/Scrapling-Skill/references/parsing/adaptive.md
index 25e2b79..5029920 100644
--- a/agent-skill/Scrapling-Skill/references/parsing/adaptive.md
+++ b/agent-skill/Scrapling-Skill/references/parsing/adaptive.md
@@ -2,7 +2,7 @@
Adaptive scraping (previously known as automatch) is one of Scrapling's most powerful features. It allows your scraper to survive website changes by intelligently tracking and relocating elements.
-Consider a page with a structure like this:
+Let's say you are scraping a page with a structure like this:
```html
<div class="container">
<section class="products">
@@ -17,7 +17,7 @@ Consider a page with a structure like this:
</section>
</div>
```
-To scrape the first product (the one with the `p1` ID), a selector like this would be used:
+And you want to scrape the first product, the one with the `p1` ID. You will probably write a selector like this
```python
page.css('#p1')
```
@@ -44,7 +44,7 @@ When website owners implement structural changes like
```
The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play.
-With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element.
+With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element, and without AI :)
```python
from scrapling import Selector, Fetcher
@@ -59,36 +59,41 @@ if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code...
```
-It works with all selection methods, not just CSS/XPath selection.
+It works with all selection methods, not just CSS/XPATH selection.
## Real-World Scenario
-This example uses [The Web Archive](https://archive.org/)'s [Wayback Machine](https://web.archive.org/) to demonstrate adaptive scraping across different versions of a website. A copy of [StackOverflow's website in 2010](https://web.archive.org/web/20100102003420/http://stackoverflow.com/) is compared against the current design to show that the adaptive feature can extract the same button using the same selector.
+Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
-To extract the Questions button from the old design, a selector like `#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a` can be used (this specific selector was generated by Chrome).
+To solve this issue, I will use [The Web Archive](https://archive.org/)'s [Wayback Machine](https://web.archive.org/). Here is a copy of [StackOverFlow's website in 2010](https://web.archive.org/web/20100102003420/http://stackoverflow.com/); pretty old, eh? Let's see if the adaptive feature can extract the same button in the old design from 2010 and the current design using the same selector :)
-Testing the same selector in both versions:
+If I want to extract the Questions button from the old design, I can use a selector like this: `#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a`. This selector is too specific because it was generated by Google Chrome.
+
+
+Now, let's test the same selector in both versions
```python
from scrapling import Fetcher
selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
new_url = "https://stackoverflow.com/"
Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')
-
page = Fetcher.get(old_url, timeout=30)
element1 = page.css(selector, auto_save=True)[0]
-
# Same selector but used in the updated website
page = Fetcher.get(new_url)
element2 = page.css(selector, adaptive=True)[0]
if element1.text == element2.text:
-... print('Scrapling found the same element in the old and new designs!')
+ print('Scrapling found the same element in the old and new designs!') # Spoiler alert: it does!
```
-The `adaptive_domain` argument is used here because Scrapling sees `archive.org` and `stackoverflow.com` as two different domains and would isolate their `adaptive` data. Passing `adaptive_domain` tells Scrapling to treat them as the same website for adaptive data storage.
+Note that I introduced a new argument called `adaptive_domain`. This is because, for Scrapling, these are two different domains (`archive.org` and `stackoverflow.com`), so Scrapling will isolate their `adaptive` data. To inform Scrapling that they are the same website, we must pass the custom domain we wish to use while saving `adaptive` data for both, ensuring Scrapling doesn't isolate them.
+
+The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
-In a typical scenario with the same URL for both requests, the `adaptive_domain` argument is not needed. The adaptive logic works the same way with both the `Selector` and `Fetcher` classes.
+Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
-**Note:** The main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, it can be used to continue using the previously stored adaptive data for the new URL. Otherwise, Scrapling will consider it a new website and discard the old data.
+**Info:**
+
+The main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, you can use it to continue using the previously stored adaptive data for the new URL. Otherwise, scrapling will consider it a new website and discard the old data.
## How the adaptive scraping feature works
Adaptive scraping works in two phases:
@@ -96,33 +101,35 @@ Adaptive scraping works in two phases:
1. **Save Phase**: Store unique properties of elements
2. **Match Phase**: Find elements with similar properties later
-After selecting an element through any method, the library can find it the next time the website is scraped, even if it undergoes structural/design changes.
+Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
-The general logic is as follows:
+With as few technical details as possible, the general logic goes as follows:
- 1. Scrapling saves that element's unique properties (methods shown below).
+ 1. You tell Scrapling to save that element's unique properties.
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
- 3. Because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. The storage system relies on two things:
- 1. The domain of the current website. When using the `Selector` class, pass it when initializing; when using a fetcher, the domain is automatically taken from the URL.
- 2. An `identifier` to query that element's properties from the database. The identifier does not always need to be set manually (see below).
+ 3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
+ 1. The domain of the current website. If you are using the `Selector` class, pass it when initializing; if you are using a fetcher, the domain will be automatically taken from the URL.
+ 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself.
Together, they will later be used to retrieve the element's unique properties from the database.
- 4. Later, when the website's structure changes, enabling `adaptive` causes Scrapling to retrieve the element's unique properties and match all elements on the page against them. A score is calculated based on their similarity to the desired element. Everything is taken into consideration in that comparison.
+ 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration.
5. The element(s) with the highest similarity score to the wanted element are returned.
### The unique properties
-The unique properties Scrapling relies on are:
+You might wonder what unique properties we are referring to when discussing the removal or alteration of all element properties.
+
+For Scrapling, the unique elements we are relying on are:
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
- Element's parent tag name, attributes (names and values), and text.
-The comparison between elements is not exact; it is based on how similar these values are. Everything is considered, including the values' order (e.g., the order in which class names are written).
+But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
## How to use adaptive feature
-The adaptive feature can be applied to any found element and is added as arguments to CSS/XPath selection methods.
+The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods.
-First, enable the `adaptive` feature by passing `adaptive=True` to the [Selector](main_classes.md#selector) class when initializing it, or enable it on the fetcher being used.
+First, you must enable the `adaptive` feature by passing `adaptive=True` to the [Selector](main_classes.md#selector) class when you initialize it or enable it in the fetcher you are using of the available fetchers.
Examples:
```python
@@ -132,39 +139,41 @@ page = Selector(html_doc, adaptive=True)
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
```
-When using the [Selector](main_classes.md#selector) class, pass the URL of the website with the `url` argument so Scrapling can separate the properties saved for each element by domain.
+If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
-If no URL is passed, the word `default` will be used in place of the URL field while saving the element's unique properties. This is only an issue when using the same identifier for a different website without passing the URL parameter. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
+If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
-The `storage` and `storage_args` arguments control the database connection; by default, the SQLite class provided by the library is used.
+Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it uses the SQLite class provided by the library. Those arguments shouldn't matter unless you want to [write your own storage system](https://scrapling.readthedocs.io/en/latest/development/adaptive_storage_system.html).
-There are two main ways to use the `adaptive` feature:
+Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
### The CSS/XPath Selection way
-First, use the `auto_save` argument while selecting an element that exists on the page:
+As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
```python
element = page.css('#p1', auto_save=True)
```
-When the element no longer exists, use the same selector with the `adaptive` argument to have the library find it:
+And when the element doesn't exist, you can use the same selector and the `adaptive` argument, and the library will find it for you
```python
element = page.css('#p1', adaptive=True)
```
-With the `css`/`xpath` methods, the identifier is set automatically to the selector string passed to the method.
+Pretty simple, eh?
+
+Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`xpath` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
### The manual way
-Elements can be manually saved, retrieved, and relocated within the `adaptive` feature. This allows relocating any element found by any method.
+You manually save and retrieve an element, then relocate it, which all happens within the `adaptive` feature, as shown below. This allows you to relocate any element using any method or selection!
-Example of getting an element by text:
+First, let's say you got an element like this by text:
```python
element = page.find_by_text('Tipping the Velvet', first_match=True)
```
-Save its unique properties using the `save` method. The identifier must be set manually (use a meaningful identifier):
+You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
```python
page.save(element, 'my_special_element')
```
-Later, retrieve and relocate the element inside the page with `adaptive`:
+Now, later, when you want to retrieve it and relocate it inside the page with `adaptive`, it would be like this
```python
>>> element_dict = page.retrieve('my_special_element')
>>> page.relocate(element_dict, selector_type=True)
@@ -172,9 +181,9 @@ Later, retrieve and relocate the element inside the page with `adaptive`:
>>> page.relocate(element_dict, selector_type=True).css('::text').getall()
['Tipping the Velvet']
```
-The `retrieve` and `relocate` methods are used here.
+Hence, the `retrieve` and `relocate` methods are used.
-To keep it as a `lxml.etree` object, omit the `selector_type` argument:
+If you want to keep it as a `lxml.etree` object, leave the `selector_type` argument
```python
>>> page.relocate(element_dict)
[<Element a at 0x105a2a7b0>]
@@ -208,4 +217,3 @@ page.save(product, 'specific_product')
## Known Issues
In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
-
diff --git a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
index 0d54498..61cd432 100644
--- a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
+++ b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md
@@ -5,7 +5,7 @@ The [Selector](#selector) class is the core parsing engine in Scrapling, providi
from scrapling import Selector
from scrapling.parser import Selector
```
-Usage:
+Then use it directly:
```python
page = Selector(
'<html>...</html>',
@@ -17,28 +17,26 @@ elements = page.css('.product')
```
In Scrapling, the main object you deal with after passing an HTML source or fetching a website is, of course, a [Selector](#selector) object. Any operation you do, like selection, navigation, etc., will return either a [Selector](#selector) object or a [Selectors](#selectors) object, given that the result is element/elements from the page, not text or similar.
-The main page is a [Selector](#selector) object, and the elements within are [Selector](#selector) objects. Any text (text content inside elements or attribute values) is a [TextHandler](#texthandler) object, and element attributes are stored as [AttributesHandler](#attributeshandler).
+In other words, the main page is a [Selector](#selector) object, and the elements within are [Selector](#selector) objects, and so on. Any text, such as the text content inside elements or the text inside element attributes, is a [TextHandler](#texthandler) object, and the attributes of each element are stored as [AttributesHandler](#attributeshandler).
## Selector
### Arguments explained
The most important one is `content`, it's used to pass the HTML code you want to parse, and it accepts the HTML content as `str` or `bytes`.
-The arguments `url`, `adaptive`, `storage`, and `storage_args` are settings used with the `adaptive` feature. They are explained in the [adaptive](adaptive.md) feature page.
+The `url` argument sets the page URL used to resolve relative links and separate saved adaptive data by domain. The `adaptive`, `storage`, and `storage_args` arguments are settings used with the [adaptive](adaptive.md) feature, and they don't make a difference if you are not going to use that feature.
-Arguments for parsing adjustments:
+Then you have the arguments for parsing adjustments or adjusting/manipulating the HTML content while the library is parsing it:
- **encoding**: This is the encoding that will be used while parsing the HTML. The default is `UTF-8`.
- **keep_comments**: This tells the library whether to keep HTML comments while parsing the page. It's disabled by default because it can cause issues with your scraping in various ways.
- **keep_cdata**: Same logic as the HTML comments. [cdata](https://stackoverflow.com/questions/7092236/what-is-cdata-in-html) is removed by default for cleaner HTML.
-The arguments `huge_tree` and `root` are advanced features not covered here.
+The `huge_tree` and `root` arguments are advanced parsing options.
-Most properties on the main page and its elements are lazily loaded (not initialized until accessed), which contributes to Scrapling's speed.
+After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
### Properties
-Properties for traversal are separated in the [traversal](#traversal) section below.
-
-Parsing this HTML page as an example:
+Let's say we are parsing this HTML page for simplicity:
```html
<html>
<head>
@@ -52,14 +50,14 @@ Parsing this HTML page as an example:
<span class="price">$10.99</span>
<div class="hidden stock">In stock: 5</div>
</article>
-
+
<article class="product" data-id="2">
<h3>Product 2</h3>
<p class="description">This is product 2</p>
<span class="price">$20.99</span>
<div class="hidden stock">In stock: 3</div>
</article>
-
+
<article class="product" data-id="3">
<h3>Product 3</h3>
<p class="description">This is product 3</p>
@@ -87,7 +85,7 @@ Get all text content on the page recursively
>>> page.get_all_text()
'Some page\n\n \n\n \nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
```
-Get the first article (used as an example throughout):
+Get the first article, as explained before; we will use it as an example
```python
article = page.find('article')
```
@@ -108,7 +106,7 @@ The `get_all_text` method has the following optional arguments:
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
-The text returned is a [TextHandler](#texthandler), not a standard string. If the text content can be serialized to JSON, use `.json()` on it:
+By the way, the text returned here is not a standard string but a [TextHandler](#texthandler); we will get to this in detail later, so if the text content can be serialized to JSON, use `.json()` on it
```python
>>> script = page.find('script')
>>> script.json()
@@ -119,7 +117,7 @@ Let's continue to get the element tag
>>> article.tag
'article'
```
-Using it on the page directly operates on the root `html` element:
+If you use it on the page directly, you will find that you are operating on the root `html` element
```python
>>> page.tag
'html'
@@ -184,9 +182,10 @@ Same case with XPath
```
### Traversal
-Properties and methods for navigating elements on the page.
+Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
-The `html` element is the root of the website's tree. Elements like `head` and `body` are "children" of `html`, and `html` is their "parent". The element `body` is a "sibling" of `head` and vice versa.
+In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.
+This element will be positioned directly above elements such as `head` and `body`. These are considered "children" of the `html` element, and the `html` element is considered their "parent". The element `body` is a "sibling" of the element `head` and vice versa.
Accessing the parent of an element
```python
@@ -195,7 +194,7 @@ Accessing the parent of an element
>>> article.parent.tag
'div'
```
-Chaining is supported, as with all similar properties/methods:
+You can chain it as you want, which applies to all similar properties/methods we will review.
```python
>>> article.parent.parent.tag
'body'
@@ -253,17 +252,17 @@ The same logic applies to the `previous` property
>>> second_article.previous
<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
```
-Check if an element has a specific class name:
+You can check easily and pretty fast if an element has a specific class name or not
```python
>>> article.has_class('product')
True
```
-Iterate over the entire ancestors' tree of any element:
+If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the example below
```python
for ancestor in article.iterancestors():
# do something with it...
```
-Search for a specific ancestor that satisfies a search function. Pass a function that takes a [Selector](#selector) object as an argument and returns `True`/`False`:
+You can search for a specific ancestor of an element that satisfies a search function; all you need to do is pass a function that takes a [Selector](#selector) object as an argument and return `True` if the condition satisfies or `False` otherwise, like below:
```python
>>> article.find_ancestor(lambda ancestor: ancestor.has_class('product-list'))
<data='<div class="product-list"> <article clas...' parent='<body> <div class="product-list"> <artic...'>
@@ -324,64 +323,43 @@ Starting with v0.4, [Selector](#selector) and [Selectors](#selectors) both provi
These methods work seamlessly with all selection types (CSS, XPath, `find`, etc.) and are the recommended way to extract text and attribute values in a Scrapy-compatible style.
+Now, let's see what [Selectors](#selectors) class adds to the table with that out of the way.
### Properties
-Apart from the standard operations on Python lists (iteration, slicing, etc.), the following operations are available:
+Apart from the standard operations on Python lists, such as iteration and slicing.
-CSS and XPath selectors can be executed directly on the [Selector](#selector) instances, with the same return types as [Selector](#selector)'s `css` and `xpath` methods. The arguments are similar, except the `adaptive` argument is not available. This makes chaining methods straightforward:
-```python
->>> page.css('.product_pod a')
-[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
-...]
+You can do the following:
->>> page.css('.product_pod').css('a') # Returns the same result
-[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
-...]
+Execute CSS and XPath selectors directly on the [Selector](#selector) instances it has, while the return types are the same as [Selector](#selector)'s `css` and `xpath` methods. The arguments are similar, except the `adaptive` argument is not available here. This, of course, makes chaining methods very straightforward.
+```pycon
+>>> page.css('.product h3::text').getall()
+['Product 1', 'Product 2', 'Product 3']
+>>> page.css('.product').css('h3::text').getall()
+['Product 1', 'Product 2', 'Product 3']
```
-The `re` and `re_first` methods can be run directly. They take the same arguments as the [Selector](#selector) class. In this class, `re_first` runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method returns a [TextHandlers](#texthandlers) object combining all matches:
-```python
->>> page.css('.price_color').re(r'[\d\.]+')
-['51.77',
- '53.74',
- '50.10',
- '47.82',
- '54.23',
-...]
+Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. See [TextHandler](#texthandler) for these methods.
->>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
-['a-light-in-the-attic_1000',
- 'tipping-the-velvet_999',
- 'soumission_998',
- 'sharp-objects_997',
-...]
+However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
+```pycon
+>>> page.css('.price::text').re(r'[\d.]+')
+['10.99', '20.99', '15.99']
+>>> page.css('.product h3::text').re(r'Product (\d+)')
+['1', '2', '3']
```
-The `search` method searches the available [Selector](#selector) instances. The function passed must accept a [Selector](#selector) instance as the first argument and return True/False. Returns the first matching [Selector](#selector) instance, or `None`:
-```python
-# Find all the products with price '53.23'.
->>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
->>> page.css('.product_pod').search(search_function)
-<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
+With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
+
+```pycon
+>>> product = page.css('.product').search(lambda p: p.css('.price::text').get() == '$20.99')
+>>> product.css('h3::text').get()
+'Product 2'
```
-The `filter` method takes a function like `search` but returns a `Selectors` instance of all matching [Selector](#selector) instances:
-```python
-# Find all products with prices over $50
->>> filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
->>> page.css('.product_pod').filter(filtering_function)
-[<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
- <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
- <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
-...]
+You can use the `filter` method, too, which takes a function like the `search` method but returns a `Selectors` instance of all the [Selector](#selector) instances that satisfy the function
+
+```pycon
+>>> products = page.css('.product').filter(lambda p: float(p.css('.price::text').re_first(r'[\d.]+')) > 15)
+>>> products.css('h3::text').getall()
+['Product 2', 'Product 3']
```
-Safe access to the first or last element without index errors:
+You can safely access the first or last element without worrying about index errors:
```python
>>> page.css('.product').first # First Selector or None
<data='<article class="product" data-id="1"><h3...'>
@@ -390,87 +368,78 @@ Safe access to the first or last element without index errors:
>>> page.css('.nonexistent').first # Returns None instead of raising IndexError
```
-Get the number of [Selector](#selector) instances in a [Selectors](#selectors) instance:
+If you are too lazy like me and want to know the number of [Selector](#selector) instances in a [Selectors](#selectors) instance. You can do this:
```python
-page.css('.product_pod').length
+page.css('.product').length
```
which is equivalent to
```python
-len(page.css('.product_pod'))
+len(page.css('.product'))
```
+Yup, like JavaScript :)
## TextHandler
-All methods/properties that return a string return `TextHandler`, and those that return a list of strings return [TextHandlers](#texthandlers) instead.
+This class is mandatory to understand, as all methods/properties that should return a string for you will return `TextHandler`, and the ones that should return a list of strings will return [TextHandlers](#texthandlers) instead.
-TextHandler is a subclass of the standard Python string, so all standard string operations are supported.
+TextHandler is a subclass of the standard Python string, so you can do anything with it that you can do with a Python string. So, what is the difference that requires a different naming?
-TextHandler provides extra methods and properties beyond standard Python strings. All methods and properties in all classes that return string(s) return TextHandler, enabling chaining and cleaner code. It can also be imported directly and used on any string.
+Of course, TextHandler provides extra methods and properties that standard Python strings can't do. All methods and properties in all classes that return string(s) return TextHandler, which opens the door for creativity and makes the code shorter and cleaner. Also, you can [import it directly and use it on any string](https://scrapling.readthedocs.io/en/latest/development/scrapling_custom_types.html).
### Usage
-All operations (slicing, indexing, etc.) and methods (`split`, `replace`, `strip`, etc.) return a `TextHandler`, so they can be chained.
+First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want.
+
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
-The `re` and `re_first` methods exist in [Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers) as well, accepting the same arguments.
+- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
-- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments but returns only the first result as a `TextHandler` instance.
-
Also, it takes other helpful arguments, which are:
-
+
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
- **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
-
- The return result is [TextHandlers](#texthandlers) because the `re` method is used:
- ```python
- >>> page.css('.price_color').re(r'[\d\.]+')
- ['51.77',
- '53.74',
- '50.10',
- '47.82',
- '54.23',
- ...]
-
- >>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
- ['a-light-in-the-attic_1000',
- 'tipping-the-velvet_999',
- 'soumission_998',
- 'sharp-objects_997',
- ...]
+
+ The return result is [TextHandlers](#texthandlers) because we used the `re` method.
+ ```pycon
+ >>> page.css('.price::text').re(r'[\d.]+')
+ ['10.99', '20.99', '15.99']
+ >>> page.css('.product h3::text').re(r'Product (\d+)')
+ ['1', '2', '3']
```
- Examples with custom strings demonstrating the other arguments:
+ To explain the other arguments better, we will use a custom string for each example below
```python
>>> from scrapling import TextHandler
>>> test_string = TextHandler('hi there') # Hence the two spaces
>>> test_string.re('hi there')
>>> test_string.re('hi there', clean_match=True) # Using `clean_match` will clean the string before matching the regex
['hi there']
-
+
>>> test_string2 = TextHandler('Oh, Hi Mark')
>>> test_string2.re_first('oh, hi Mark')
>>> test_string2.re_first('oh, hi Mark', case_sensitive=False) # Hence disabling `case_sensitive`
'Oh, Hi Mark'
-
+
# Mixing arguments
>>> test_string.re('hi there', clean_match=True, case_sensitive=False)
- ['hi There']
+ ['hi there']
```
- Since `html_content` returns `TextHandler`, regex can be applied directly on HTML content:
+ Another use of the idea of replacing strings with `TextHandler` everywhere is that a property like `html_content` returns `TextHandler`, so you can do regex on the HTML content if you want:
```python
>>> page.html_content.re('div class=".*">(.*)</div')
['In stock: 5', 'In stock: 3', 'Out of stock']
```
-- The `.json()` method converts the content to a JSON object if possible; otherwise, it throws an error:
+- You also have the `.json()` method, which tries to convert the content to a JSON object quickly if possible; otherwise, it throws an error
```python
>>> page.css('#page-data::text').get()
'\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n '
>>> page.css('#page-data::text').get().json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
- If no text node is specified while selecting an element, the text content is selected automatically:
+ Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically, like this
```python
>>> page.css('#page-data')[0].json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
- The [Selector](#selector) class adds additional behavior. Given this page:
+ The [Selector](#selector) class adds one thing here, too; let's say this is the page we are working with:
```html
<html>
<body>
@@ -485,41 +454,43 @@ The `re` and `re_first` methods exist in [Selector](#selector), [Selectors](#sel
</body>
</html>
```
- The [Selector](#selector) class has the `get_all_text` method, which returns a `TextHandler`. For example:
+ The [Selector](#selector) class has the `get_all_text` method, which you should be aware of by now. This method returns a `TextHandler`, of course.
+ So, as you know here, if you did something like this
```python
>>> page.css('div::text').get().json()
```
- This throws an error because the `div` tag has no direct text content. The `get_all_text` method handles this case:
+ You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.
+ In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python
>>> page.css('div')[0].get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
- The `ignore_tags` argument is used here because its default value is `('script', 'style',)`.
-
- When dealing with a JSON response:
+ I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
+ Another related behavior to be aware of occurs when using any fetcher. If you have a JSON response like this example:
```python
>>> page = Selector("""{"some_key": "some_value"}""")
```
- The [Selector](#selector) class is optimized for HTML, so it treats this as a broken HTML response and wraps it. The `html_content` property shows:
+ Because the [Selector](#selector) class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the `html_content` property, you get this
```python
>>> page.html_content
'<html><body><p>{"some_key": "some_value"}</p></body></html>'
```
- The `json` method can be used directly:
+ Here, you can use the `json` method directly, and it will work
```python
>>> page.json()
{'some_key': 'some_value'}
```
- For JSON responses, the [Selector](#selector) class keeps a raw copy of the content it receives. When `.json()` is called, it checks for that raw copy first and converts it to JSON. If the raw copy is unavailable (as with sub-elements), it checks the current element's text content, then falls back to `get_all_text`.
+ You might wonder how this happened, given that the `html` tag doesn't contain direct text.
+ Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is unavailable, as with the elements, it checks the current element's text content; otherwise, it uses the `get_all_text` method directly.
-- The `.clean()` method removes all whitespace and consecutive spaces, returning a new `TextHandler` instance:
+- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
```
-The `remove_entities` argument causes `clean` to replace HTML entities with their corresponding characters.
+Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
-- The `.sort()` method sorts the string characters:
+- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python
>>> TextHandler('acb').sort()
'abc'
@@ -533,9 +504,9 @@ Or do it in reverse:
This class is returned in place of strings nearly everywhere in the library.
## TextHandlers
-This class inherits from standard lists, adding `re` and `re_first` as new methods.
+You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
-The `re_first` method runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`.
+The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`.
## AttributesHandler
This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element/[Selector](#selector) instance.
@@ -551,8 +522,8 @@ It currently adds two extra simple methods:
- The `search_values` method
- Searches the current attributes by values (rather than keys) and returns a dictionary of each matching item.
-
+ In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
+
A simple example would be
```python
>>> for i in page.find('script').attrib.search_values('page-data'):
@@ -565,7 +536,7 @@ It currently adds two extra simple methods:
print(i)
{'id': 'page-data'}
```
- A more practical example is using it with `find_all` to find all elements that have a specific value in their attributes:
+ These examples won't happen in the real world; most likely, a more real-world example would be using it with the `find_all` method to find all elements that have a specific value in their arguments:
```python
>>> page.find_all(lambda element: list(element.attrib.search_values('product')))
[<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>,
@@ -573,14 +544,14 @@ It currently adds two extra simple methods:
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
```
All these elements have 'product' as the value for the `class` attribute.
-
- The `list` function is used here because `search_values` returns a generator, so it would be `True` for all elements.
+
+ Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
-
+
```python
>>>page.find('script').attrib.json_string
b'{"id":"page-data","type":"application/json"}'
- ```
\ No newline at end of file
+ ```
diff --git a/agent-skill/Scrapling-Skill/references/parsing/selection.md b/agent-skill/Scrapling-Skill/references/parsing/selection.md
index d82a802..7bccd4b 100644
--- a/agent-skill/Scrapling-Skill/references/parsing/selection.md
+++ b/agent-skill/Scrapling-Skill/references/parsing/selection.md
@@ -1,5 +1,5 @@
# Querying elements
-Scrapling currently supports parsing HTML pages exclusively (no XML feeds), because the adaptive feature does not work with XML.
+The `Selector` class parses HTML. This decision was made because the adaptive feature won't work with XML. For XML, RSS, or Atom feeds, use [XMLFeedSpider](../spiders/generic-templates.md#xmlfeedspider), which uses an XML parser and gives you each feed node directly.
In Scrapling, there are five main ways to find elements:
@@ -9,7 +9,7 @@ In Scrapling, there are five main ways to find elements:
4. Finding elements whose content contains a specific text
5. Finding elements whose content matches a specific regex
-There are also other indirect ways to find elements. Scrapling can also find elements similar to a given element; see [Finding Similar Elements](#finding-similar-elements).
+One of the most remarkable features of Scrapling is the ability to [find elements that are similar to the element you have](#finding-similar-elements).
## CSS/XPath selectors
@@ -23,18 +23,18 @@ Also, Scrapling implements some non-standard pseudo-elements like:
* To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
-The selector logic follows the same conventions as Scrapy/Parsel.
+In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
To select elements with CSS selectors, use the `css` method, which returns `Selectors`. Use `[0]` to get the first element, or `.get()` / `.getall()` to extract text values from text/attribute pseudo-selectors.
### What are XPath selectors?
[XPath](https://en.wikipedia.org/wiki/XPath) is a language for selecting nodes in XML documents, which can also be used with HTML. This [cheatsheet](https://devhints.io/xpath) is a good resource for learning about [XPath](https://en.wikipedia.org/wiki/XPath). Scrapling adds XPath selectors directly through [lxml](https://lxml.de/).
-The logic follows the same conventions as Scrapy/Parsel. However, Scrapling does not implement the XPath extension function `has-class` as Scrapy/Parsel does. Instead, it provides the `has_class` method on returned elements.
+In short, it is the same situation as CSS Selectors; if you come from Scrapy/Parsel, you will find the same logic for selectors here. However, Scrapling doesn't implement the XPath extension function `has-class` as Scrapy/Parsel does. Instead, it provides the `has_class` method, which can be used on elements returned for the same purpose.
-To select elements with XPath selectors, use the `xpath` method, which follows the same logic as the CSS selectors method above.
+To select elements with XPath selectors, you have the `xpath` method. Again, this method follows the same logic as the CSS selectors method above.
-> Note that each method of `css` and `xpath` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
+> The `css` and `xpath` methods have additional arguments for the [adaptive feature](adaptive.md).
### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors.
@@ -44,7 +44,9 @@ Select all elements with the class `product`.
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
```
-**Note:** The XPath version won't be accurate if there's another class; it's always better to rely on CSS for selecting by class.
+**Note:**
+
+The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
Select the first element with the class `product`.
```python
@@ -74,9 +76,11 @@ title = page.xpath('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
You can nest and chain selectors as you want, given that they return results
```python
page.css('.product')[0].css('h1:contains("Phone")::text').get()
-page.xpath('//*[@class="product"]')[0].xpath('//h1[contains(text(),"Phone")]/text()').get()
+page.xpath('//*[@class="product"]')[0].xpath('.//h1[contains(text(),"Phone")]/text()').get()
page.xpath('//*[@class="product"]')[0].css('h1:contains("Phone")::text').get()
```
+Use `.//` for an XPath relative to the current element. A leading `//` searches the whole document, even after selecting a product.
+
Another example
All links that have 'image' in their 'href' attribute
@@ -90,12 +94,12 @@ for index, link in enumerate(links):
```
## Text-content selection
-Scrapling provides two ways to select elements based on their direct text content:
+Scrapling provides the ability to select elements based on their direct text content, and you have two ways to do this:
1. Elements whose direct text content contains the given text with many options through the `find_by_text` method.
2. Elements whose direct text content matches the given regex pattern with many options through the `find_by_regex` method.
-Anything achievable with `find_by_text` can also be done with `find_by_regex`, but both are provided for convenience.
+What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
@@ -107,25 +111,33 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
* **partial**: If enabled, `find_by_text` will return elements that contain the input text. So it's not an exact match anymore
-**Note:** The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument.
+**Note:**
+
+The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements
-Scrapling can find elements similar to a given element, inspired by the AutoScraper library but usable with elements found by any method.
+One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
-Given an element (e.g., a product found by title), calling `.find_similar()` on it causes Scrapling to:
+So, how does it work?
-1. Find all page elements with the same DOM tree depth as this element.
+Imagine a scenario where you found a product by its title, for example, and you want to extract other products listed in the same table/container. With the element you have, you can call the method `.find_similar()` on it, and Scrapling will:
+
+1. Find all page elements with the same DOM tree depth as this element.
2. All found elements will be checked, and those without the same tag name, parent tag name, and grandparent tag name will be dropped.
-3. As a final check, Scrapling uses fuzzy matching to drop elements whose attributes don't resemble the original element's attributes. A configurable percentage controls this step (see arguments below).
+3. Now we are sure (like 99% sure) that these elements are the ones we want, but as a last check, Scrapling will use fuzzy matching to drop the elements whose attributes don't look like the attributes of our element. There's a percentage to control this step, and I recommend you not play with it unless the default settings don't get the elements you want.
-Arguments for `find_similar()`:
+These are the arguments that can be passed to this method:
-* **similarity_threshold**: The percentage for comparing elements' attributes (step 3). Default is 0.2 (tag attributes must be at least 20% similar). Set to 0 to disable this check entirely.
+* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0.
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
+Now, let's check out the examples below.
+
### Examples
-Examples of finding elements with raw text, regex, and `find_similar`.
+Let's see some shared examples of finding elements with raw text and regex.
+
+These examples use the `Fetcher` class.
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://books.toscrape.com/index.html')
@@ -161,7 +173,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
```
-The search is case-insensitive by default, so those results include `The`, not just the lowercase `the`. To limit to exact case:
+The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
@@ -206,7 +218,7 @@ Find all elements similar to the current element in location and attributes. For
<data='<a href="catalogue/sharp-objects_997/ind...' parent='<h3><a href="catalogue/sharp-objects_997...'>,
...]
```
-The number of elements is 19, not 20, because the current element is not included in the results:
+Notice that the number of elements is 19, not 20, because the current element is not included in the results.
```python
>>> len(element.find_similar(ignore_attributes=['title']))
19
@@ -222,21 +234,23 @@ Get the `href` attribute from all similar elements
'catalogue/sharp-objects_997/index.html',
...]
```
-Getting all books' data using that element as a starting point:
+To increase the complexity a little bit, let's say we want to get all the books' data using that element as a starting point for some reason
```python
->>> for product in element.parent.parent.find_similar():
+>>> first_product = element.parent.parent
+>>> for product in [first_product, *first_product.find_similar()]:
print({
"name": product.css('h3 a::text').get(),
"price": product.css('.price_color')[0].re_first(r'[\d\.]+'),
"stock": product.css('.availability::text').getall()[-1].clean()
})
+{'name': 'Tipping the Velvet', 'price': '53.74', 'stock': 'In stock'}
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
...
```
### Advanced examples
-Advanced examples using the `find_similar` method:
+See more advanced or real-world examples using the `find_similar` method.
E-commerce Product Extraction
```python
@@ -247,13 +261,13 @@ def extract_product_grid(page):
)
# Find similar product cards
- products = first_product.find_similar()
+ products = [first_product, *first_product.find_similar()]
return [
{
'name': p.css('h3::text').get(),
'price': p.css('.price::text').re_first(r'\d+\.\d{2}'),
- 'stock': 'In stock' in p.text,
+ 'stock': 'In stock' in p.get_all_text(),
'rating': p.css('.rating')[0].attrib.get('data-rating')
}
for p in products
@@ -266,7 +280,7 @@ def extract_table_data(page):
first_row = page.css('table tbody tr')[0]
# Find similar rows
- rows = first_row.find_similar()
+ rows = [first_row, *first_row.find_similar()]
return [
{
@@ -286,7 +300,7 @@ def extract_form_fields(page):
)
# Find similar field containers
- fields = first_field.find_similar()
+ fields = [first_field, *first_field.find_similar()]
return [
{
@@ -305,10 +319,10 @@ def extract_reviews(page):
review_container = first_review.find_ancestor(
lambda e: e.has_class('review')
)
-
+
# Find similar reviews
- all_reviews = review_container.find_similar()
-
+ all_reviews = [review_container, *review_container.find_similar()]
+
return [
{
'text': r.css('.review-text::text').get(),
@@ -319,7 +333,9 @@ def extract_reviews(page):
]
```
## Filters-based searching
-Inspired by BeautifulSoup's `find_all` function, elements can be found using the `find_all` and `find` methods. Both methods accept multiple filters and return all elements on the pages where all filters apply.
+This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
+
+Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific:
@@ -341,8 +357,10 @@ It filters all elements in the current page/element in the following order:
**Notes:**
-1. The filtering process always starts from the first filter it finds in the filtering order above. If no tag name(s) are passed but attributes are passed, the process starts from step 2, and so on.
-2. The order in which arguments are passed does not matter. The only order considered is the one explained above.
+1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
+2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
+
+Check examples to clear any confusion :)
### Examples
```python
@@ -433,7 +451,7 @@ Another pro tip: Find all elements whose `href` attribute's value has '/author/'
And so on...
## Generating selectors
-CSS/XPath selectors can be generated for any element, regardless of the method used to find it.
+You can always generate CSS/XPath selectors for any element that can be reused here or anywhere else, and the most remarkable thing is that it doesn't matter what method you used to find that element!
Generate a short CSS selector for the `url_element` element (if possible, create a short one; otherwise, it's a full selector)
```python
@@ -456,12 +474,14 @@ Generate a full XPath selector for the `url_element` element from the start of t
>>> url_element.generate_full_xpath_selector
'//body/div/div[2]/div/div/span[2]/a'
```
-**Note:** When generating a short selector, Scrapling tries to find a unique element (e.g., one with an `id` attribute) as a stop point. If none exists, the short and full selectors will be identical.
+**Note:**
+
+When you tell Scrapling to create a short selector, it tries to find a unique element to use in generation as a stop point, like an element with an `id` attribute, but in our case, there wasn't any, so that's why the short and the full selector will be the same.
## Using selectors with regular expressions
-Similar to `parsel`/`scrapy`, `re` and `re_first` methods are available for extracting data using regular expressions. These methods exist in `Selector`, `Selectors`, `TextHandler`, and `TextHandlers`, so they can be used directly on elements even without selecting a text node. See the [TextHandler](main_classes.md#texthandler) class for details.
+Similar to `parsel`/`scrapy`, `re` and `re_first` methods are available for extracting data using regular expressions. However, unlike the former libraries, these methods are in nearly all classes like `Selector`/`Selectors`/`TextHandler` and `TextHandlers`, which means you can use them directly on the element even if you didn't select a text node.
-Examples:
+See [TextHandler](main_classes.md#texthandler) for details on these methods.
```python
>>> page.css('.price_color')[0].re_first(r'[\d\.]+')
'51.77'
@@ -491,4 +511,3 @@ Examples:
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
```
-See the [TextHandler](main_classes.md#texthandler) class for more details on regex methods.
\ No newline at end of file
diff --git a/agent-skill/Scrapling-Skill/references/spiders/advanced.md b/agent-skill/Scrapling-Skill/references/spiders/advanced.md
index bdb2333..ea4aeff 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/advanced.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/advanced.md
@@ -9,7 +9,7 @@ The spider system uses three class attributes to control how aggressively it cra
| `concurrent_requests` | `4` | Maximum number of requests being processed at the same time |
| `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) |
| `download_delay` | `0.0` | Seconds to wait before each request |
-| `robots_txt_obey` | `False` | Respect robots.txt rules (Disallow, Crawl-delay, Request-rate) |
+| `robots_txt_obey` | `False` | Respect robots.txt rules (Disallow, Crawl-delay, Request-rate) |
```python
class PoliteSpider(Spider):
@@ -25,9 +25,11 @@ class PoliteSpider(Spider):
yield {"title": response.css("title::text").get("")}
```
-When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously - you can allow high global concurrency while being polite to each individual domain.
+When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously, as you can allow high global concurrency while being polite to each individual domain.
-**Tip:** The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.
+**Tip:**
+
+The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.
## AutoThrottle
@@ -82,13 +84,13 @@ print(result.stats.autothrottle_delays) # {'example.com': 0.62}
**Notes:**
-- `download_delay` and any `Crawl-delay` from robots.txt act as a floor. AutoThrottle only ever adjusts the delay above them, so politeness settings are never undercut.
-- `concurrent_requests_per_domain` does double duty here: it caps how many requests may be in flight to a domain and doubles as AutoThrottle's target, so there's usually no separate target to configure. Setting it is recommended, because while a throttled domain sleeps it holds a slot of the global limiter, and leaving the per-domain limit unlimited means those sleeping slots come out of the budget shared with every other domain.
-- The measured latency is the full fetch, so it includes any internal retries and, for browser sessions, the page render time.
-- `autothrottle_max_delay` caps everything, including `Retry-After`. If a website asks for longer than your ceiling, raise `autothrottle_max_delay` to honor it.
-- The learned delays are not checkpointed. After a pause and resume, each domain starts again from `autothrottle_start_delay`.
+* `download_delay` and robots.txt delays act as a floor, but `autothrottle_max_delay` also caps that floor. Set the maximum at least as high as the configured or robots.txt delay you need to honor.
+* `concurrent_requests_per_domain` does double duty here: it caps how many requests may be in flight to a domain and doubles as AutoThrottle's target, so there's usually no separate target to configure. Setting it is recommended, because while a throttled domain sleeps it holds a slot of the global limiter, and leaving the per-domain limit unlimited means those sleeping slots come out of the budget shared with every other domain.
+* The measured latency is the full fetch, so it includes any internal retries and, for browser sessions, the page render time.
+* `autothrottle_max_delay` caps everything, including `Retry-After`. If a website asks for longer than your ceiling, raise `autothrottle_max_delay` to honor it.
+* The learned delays are not checkpointed. After a pause and resume, each domain starts again from `autothrottle_start_delay`.
-### Using uvloop
+## Using uvloop
The `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available:
@@ -119,7 +121,7 @@ else:
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`.
4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.
-**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**
+**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**
You can change the interval as follows:
@@ -130,7 +132,9 @@ spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)
The writing to the disk is atomic, so it's totally safe.
-**Tip:** Pressing `Ctrl+C` during a crawl always causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.
+**Tip:**
+
+With `start()`, pressing `Ctrl+C` during a crawl causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.
### Knowing If You're Resuming
@@ -146,7 +150,7 @@ async def on_start(self, resuming: bool = False):
## Development Mode
-When you're iterating on a spider's `parse()` logic, re-hitting the target servers on every run is slow and noisy. Development mode caches every response to disk on the first run and replays them from disk on subsequent runs, so you can tweak your selectors and re-run the spider as many times as you want without making a single network request.
+When you're iterating on a spider's `parse()` logic, re-hitting the target servers on every run is slow and noisy. Development mode caches every response to disk on the first run and replays them from disk on subsequent runs, so you can tweak your selectors and re-run the spider as many times as you want without fetching those pages again.
Enable it by setting `development_mode = True` on your spider:
@@ -160,7 +164,7 @@ class MySpider(Spider):
yield {"title": response.css("title::text").get("")}
```
-The first run fetches normally and stores each response on disk. Every subsequent run serves the same requests from the cache, skipping the network entirely.
+The first run fetches normally and stores each response on disk. Every subsequent run serves the same requests from the cache, skipping the page fetch. With `robots_txt_obey=True`, robots.txt checks run before cache lookup.
### Cache Location
@@ -178,14 +182,16 @@ class MySpider(Spider):
1. **Cache key**: Each response is keyed by the request's fingerprint, so any change to fingerprint-affecting attributes (`fp_include_kwargs`, `fp_include_headers`, `fp_keep_fragments`) will produce a fresh fetch.
2. **Storage format**: One JSON file per response, named `{fingerprint_hex}.json`. The body is base64-encoded so binary content is preserved exactly. Writes are atomic (temp file + rename).
-3. **Replay**: On a cache hit, the engine skips the network entirely, including `download_delay`, rate limiting, and the `is_blocked()` retry path. The cached response goes straight to your callback.
+3. **Replay**: On a cache hit, the engine skips the page fetch, including `download_delay`, rate limiting, and the `is_blocked()` retry path. The cached response goes straight to your callback.
4. **Stats**: Cached requests still count toward `requests_count`, `response_bytes`, and the per-status counters, so your stat output looks the same as a normal crawl. Two extra counters, `cache_hits` and `cache_misses`, let you see how the cache performed.
### Clearing the Cache
There's no automatic expiration. To force a fresh crawl, delete the cache directory or call the manager's `clear()` method directly.
-**Warning:** Development mode is meant for development, not production. Cached responses never expire, and replay bypasses rate limiting and blocked-request retries. Don't ship a spider with `development_mode = True`.
+**Warning:**
+
+Development mode is meant for development, not production. Cached responses never expire, and replay bypasses rate limiting and blocked-request retries. Don't ship a spider with `development_mode = True`.
## Streaming
@@ -211,7 +217,9 @@ Key differences from `start()`:
- Items are yielded one by one as they're scraped, not collected into a list
- You can access `spider.stats` during iteration for real-time statistics
-**Note:** The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results-statistics).
+**Abstract:**
+
+The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results-statistics)
You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed.
@@ -279,7 +287,9 @@ async def on_scraped_item(self, item: dict) -> dict | None:
return item
```
-**Tip:** This hook can also be used to direct items through your own pipelines and drop them from the spider.
+**Tip:**
+
+This hook can also be used to direct items through your own pipelines and drop them from the spider.
### start_requests
@@ -404,4 +414,4 @@ class MySpider(Spider):
yield {"title": response.css("title::text").get("")}
```
-The log file directory is created automatically if it doesn't exist. Both console and file output use the same format.
\ No newline at end of file
+The log file directory is created automatically if it doesn't exist. Both console and file output use the same format.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
index 11fd8ef..8d640d8 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
@@ -1,12 +1,10 @@
# Spiders architecture
-Scrapling's spider system is an async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.
+Scrapling's spider system is a Scrapy-inspired async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.
## Data Flow
-The diagram below shows how data flows through the spider system when a crawl is running:
-
-Here's what happens step by step when you run a spider:
+Here's what happens step by step when you run a spider without many details:
1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic.
2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first.
@@ -90,4 +88,4 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:
| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()` or custom through hooks |
| Running | `scrapy crawl spider_name` | `MySpider().start()` |
| Streaming | N/A | `async for item in spider.stream()` |
-| Multi-session | N/A | Multiple sessions with different types per spider |
\ No newline at end of file
+| Multi-session | N/A | Multiple sessions with different types per spider |
diff --git a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
index df465ef..83e368d 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
@@ -1,8 +1,8 @@
# Generic Spider Templates
-Most crawls fall into one of two patterns: "follow links matching this regex" or "crawl every URL listed in the site's sitemap". Scrapling ships templates for both so you don't have to hand-write the same `parse()` boilerplate every time.
+Most crawls fall into one of two patterns: "follow links matching this pattern" or "crawl every URL listed in the site's sitemap". Scrapling ships templates for both so you don't have to hand-write the same `parse()` boilerplate every time.
-Both templates build on `LinkExtractor`, which pulls URLs out of a `Response` (or filters a single URL via `matches()`). `SitemapSpider` additionally parses sitemap.xml / sitemap_index.xml bodies internally (gzip-compressed or not).
+`CrawlSpider` and `SitemapSpider` build on `LinkExtractor`, which pulls URLs out of a `Response` (or filters a single URL via `matches()`). `SitemapSpider` additionally parses sitemap.xml / sitemap_index.xml bodies internally (gzip-compressed or not).
You can use `LinkExtractor` directly inside any plain `Spider.parse()`. The templates just save you the wiring.
@@ -13,28 +13,29 @@ You can use `LinkExtractor` directly inside any plain `Spider.parse()`. The temp
```python
from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor
-class BlogCrawler(CrawlSpider):
+class QuotesSpider(CrawlSpider):
name = "blog"
- start_urls = ["https://example.com"]
+ start_urls = ["https://quotes.toscrape.com/"]
def rules(self):
return [
- CrawlRule(LinkExtractor(allow=r"/posts/"), callback=self.parse_post),
+ CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author),
CrawlRule(LinkExtractor(allow=r"/page/\d+/")), # follow pagination, no callback
]
- async def parse_post(self, response):
+ async def parse_author(self, response):
yield {
- "title": response.css("h1::text").get(),
+ '.author-title': response.css('.author-title::text').get(),
+ "birthday": response.css('.author-born-date::text').get(),
"url": response.url,
}
-result = BlogCrawler().start()
+result = QuotesSpider().start()
```
A `CrawlRule` pairs a `LinkExtractor` with an optional `callback` (a bound method on the spider), an optional `priority` override for the dispatched `Request`, and an optional `process_request` (a bound method that mutates each `Request` before it's yielded). The default `parse()` runs every rule against every response and yields a `Request` per matched URL.
-If a rule has no callback, the matched URLs fall through to the spider's default `parse()` (or stay uncallback'd if you didn't override it). This is convenient for pagination: extract the next-page links to keep the crawl going, but don't need a separate handler.
+If a rule has no callback, the matched URLs fall through to the spider's default `parse()`. This is convenient for pagination: extract the next-page links to keep the crawl going, without needing a separate handler.
### Crawl order
@@ -109,7 +110,7 @@ result = MySitemap().start()
### How URLs are dispatched
-For each URL in the sitemap, `SitemapSpider` checks every rule's `LinkExtractor.matches(url)` in order. The first matching rule wins, and a `Request` is yielded with that rule's callback. If no rule matches and `rules()` is non-empty, the URL is dropped (matches Scrapy's behavior). If `rules()` returns an empty list, every URL is routed to the spider's `parse()` method, which raises `NotImplementedError` by default - override it to handle them.
+For each URL in the sitemap, `SitemapSpider` checks every rule's `LinkExtractor.matches(url)` in order. The first matching rule wins, and a `Request` is yielded with that rule's callback. If no rule matches and `rules()` is non-empty, the URL is dropped. If `rules()` returns an empty list, every URL is routed to the spider's `parse()` method, which raises `NotImplementedError` by default if not overridden.
### Sitemap indexes
@@ -124,12 +125,12 @@ class MySitemap(SitemapSpider):
### Robots.txt support
-Put a `robots.txt` URL directly in `sitemap_urls` and `SitemapSpider` will detect it, extract every `Sitemap:` directive (via `protego`), and follow each one:
+Put a `robots.txt` URL directly in `sitemap_urls` and `SitemapSpider` will detect it, extract every sitemap shown, and follow each one:
```python
class MySitemap(SitemapSpider):
name = "sm"
- sitemap_urls = ["https://example.com/robots.txt"] # Sitemap: directives discovered automatically
+ sitemap_urls = ["https://example.com/robots.txt"]
```
### Alternate-language URLs
@@ -208,7 +209,7 @@ Gzipped feeds are decompressed automatically here as well, as shown above for **
## ContentSpider
-`ContentSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page (`url`, `title`, `markdown` keys) and optionally writing one Markdown file per page:
+`ContentSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page and optionally writing one Markdown file per page:
```python
from scrapling.spiders import ContentSpider
@@ -222,7 +223,7 @@ class DocsSpider(ContentSpider):
result = DocsSpider().start()
```
-It requires `allowed_domains` to keep the crawl bound to the target website. Options: `css_selector`/`main_content_only` (passed to `markdown()` per page, `main_content_only` enabled by default), `output_dir`, and `max_pages` (max pages to convert, `0` disables). Every page link inside `allowed_domains` is followed by default; override `rules()` with a `LinkExtractor` (`allow` to narrow the crawl, `deny` to drop URL patterns) to control it. The full guide is in `../building-rag-systems.md`.
+It requires `allowed_domains` to keep the crawl bound to the target website. The full guide, including all its options and how to feed the output to RAG pipelines, is on the [Building RAG systems](../building-rag-systems.md) page.
## Using `LinkExtractor` directly
@@ -249,21 +250,21 @@ class CustomSpider(Spider):
## LinkExtractor reference
-| Argument | Default | Description |
-|---|---|---|
-| `allow` | `()` | URL patterns to keep. Empty means "match all". String, compiled `Pattern`, or iterable of either. |
-| `deny` | `()` | URL patterns to drop. Always overrides `allow`. |
-| `allow_domains` | `()` | Hostnames to keep. Subdomains match automatically (`example.com` matches `api.example.com`). |
-| `deny_domains` | `()` | Hostnames to drop. |
-| `restrict_css` | `()` | CSS selectors that scope DOM extraction to a region. |
-| `restrict_xpath` | `()` | XPath selectors that scope DOM extraction to a region. |
-| `tags` | `("a", "area")` | Element tags to look for links in. |
-| `attrs` | `("href",)` | Attributes on those tags to read URLs from. |
-| `canonicalize` | `True` | Sort query params and normalize the path. |
-| `strip` | `True` | Strip whitespace from extracted URLs. |
-| `keep_fragment` | `False` | Preserve the `#fragment` when canonicalizing. |
-| `deny_extensions` | `IGNORED_EXTENSIONS` | File extensions to drop (pdf, zip, images, video, etc.). |
-| `process` | `None` | Optional callable applied to each extracted URL before filtering. Return a falsy value to drop. |
+| Argument | Default | Description |
+|-------------------|----------------------|---------------------------------------------------------------------------------------------------|
+| `allow` | `()` | URL patterns to keep. Empty means "match all". String, compiled `Pattern`, or iterable of either. |
+| `deny` | `()` | URL patterns to drop. Always overrides `allow`. |
+| `allow_domains` | `()` | Hostnames to keep. Subdomains match automatically (`example.com` matches `api.example.com`). |
+| `deny_domains` | `()` | Hostnames to drop. |
+| `restrict_css` | `()` | CSS selectors that scope DOM extraction to a region. |
+| `restrict_xpath` | `()` | XPath selectors that scope DOM extraction to a region. |
+| `tags` | `("a", "area")` | Element tags to look for links in. |
+| `attrs` | `("href",)` | Attributes on those tags to read URLs from. |
+| `canonicalize` | `True` | Sort query params and normalize the path. |
+| `strip` | `True` | Strip whitespace from extracted URLs. |
+| `keep_fragment` | `False` | Preserve the `#fragment` when canonicalizing. |
+| `deny_extensions` | `IGNORED_EXTENSIONS` | File extensions to drop (pdf, zip, images, video, etc.). |
+| `process` | `None` | Optional callable applied to each extracted URL before filtering. Return a falsy value to drop. |
`LinkExtractor.extract(response)` returns a `list[str]` of absolute, filtered, deduped URLs.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
index f18073e..f7d4df9 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
@@ -1,5 +1,9 @@
# Getting started
+## Introduction
+
+The spider system lets you build concurrent, multi-page crawlers in just a few lines of code.
+
## Your First Spider
A spider is a class that defines how to crawl and extract data from websites. Here's the simplest possible spider:
@@ -21,11 +25,11 @@ class QuotesSpider(Spider):
Every spider needs three things:
-1. **`name`**: A unique identifier for the spider.
-2. **`start_urls`**: A list of URLs to start crawling from.
-3. **`parse()`**: An async generator method that processes each response and yields results.
+1. **`name`** - A unique identifier for the spider.
+2. **`start_urls`** - A list of URLs to start crawling from.
+3. **`parse()`** - An async generator method that processes each response and yields results.
-The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
+The `parse()` method is where the magic happens. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
## Running the Spider
@@ -35,7 +39,7 @@ To run your spider, create an instance and call `start()`:
result = QuotesSpider().start()
```
-The `start()` method handles all the async machinery internally, so there is no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
+The `start()` method handles all the async machinery internally, so no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
Those stats are in the returned `CrawlResult` object, which gives you everything you need:
@@ -96,7 +100,9 @@ async def parse_product(self, response: Response):
}
```
-**Note:** All callback methods must be async generators (using `async def` and `yield`).
+**Note:**
+
+All callback methods must be async generators (using `async def` and `yield`).
## Exporting Data
@@ -164,11 +170,10 @@ When enabled, the spider will:
1. **Pre-fetch robots.txt** for all domains in `start_urls` before the crawl begins (concurrently).
2. **Check every request** against the domain's robots.txt `Disallow` rules. Disallowed requests are silently dropped and counted in `stats.robots_disallowed_count`.
-3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed.
+3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed. When AutoThrottle is enabled, its maximum delay also caps this value. Set `autothrottle_max_delay` high enough to honor the required delay.
Robots.txt files are fetched using the spider's default session and cached per domain for the entire crawl. Domains discovered mid-crawl (not in `start_urls`) have their robots.txt fetched on the first request to that domain.
-**Note:** `robots_txt_obey` is turned off by default. It does not affect your concurrency settings -- only the delay between requests is adjusted.
+**Note:** `robots_txt_obey` is turned off by default to avoid surprising behavior. If you enable it, it does not affect your concurrency settings (`concurrent_requests`, `concurrent_requests_per_domain`) -- only the delay between requests is adjusted.
If you don't want to pick a `download_delay` by hand, set `autothrottle_enabled = True` and the spider will tune the delay of every domain on its own from how fast it responds, backing off when it gets blocked. See [AutoThrottle](advanced.md#autothrottle).
-
diff --git a/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md b/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
index 8f67f67..2f21cd8 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
@@ -48,7 +48,3 @@ These fields are not mandatory; override the `_process_product()` method in your
- The JSON endpoints only expose products published to the online-store channel, so a collection's `products_count` can be higher than what's actually retrievable. The spider treats it as "nonzero means fetch this collection", never as an expected total.
- This template most likely won't work on stores behind extra protections or password-protected ones, and if it can work at all, it will need a lot of overrides from your side.
- Everything from `Spider` still applies: concurrency settings, delays, robots.txt compliance, checkpoints, and the lifecycle hooks.
-
-### What qualifies as a platform template
-
-Platform templates are accepted for platforms that expose a uniform, machine-readable structure across many independent websites, like Shopify does. Spiders for one specific website don't belong in the library, no matter how popular the website is.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
index 536b7f0..0c08fb0 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
@@ -1,6 +1,8 @@
# Proxy management and handling Blocks
-Scrapling's `ProxyRotator` manages proxy rotation across requests. It works with all session types and integrates with the spider's blocked request retry system.
+## Introduction
+
+When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward. It works with all session types and integrates with the spider's blocked request retry system.
## ProxyRotator
@@ -31,7 +33,7 @@ class MySpider(Spider):
Each request automatically gets the next proxy in the rotation. The proxy used is stored in `response.meta["proxy"]` so you can track which proxy fetched which page.
-Browser sessions support both string and dict proxy formats:
+When you use it with browser sessions, you will need some adjustments, like below:
```python
from scrapling.fetchers import AsyncStealthySession, ProxyRotator
@@ -54,14 +56,14 @@ def configure_sessions(self, manager):
manager.add("browser", AsyncStealthySession(proxy_rotator=rotator))
```
-**Important:**
+**Info:**
-1. You cannot use the `proxy_rotator` argument together with the static `proxy` or `proxies` parameters on the same session. Pick one approach when configuring the session, and override it per request later if needed.
-2. By default, all browser-based sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
+1. You cannot use the `proxy_rotator` argument together with the static `proxy` or `proxies` parameters on the same session. Pick one approach when configuring the session, and override it per request later if you want.
+2. Remember that by default, all browser-based sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
## Custom Rotation Strategies
-By default, `ProxyRotator` uses cyclic rotation - it iterates through proxies sequentially, wrapping around at the end.
+By default, `ProxyRotator` uses cyclic rotation, iterating through proxies sequentially and wrapping around at the end.
You can provide a custom strategy function to change this behavior, but it has to match the below signature:
@@ -232,4 +234,4 @@ class MySpider(Spider):
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
-The above logic is: requests are made with cheap proxies, such as datacenter proxies, until they are blocked, then retried with higher-quality proxies, such as residential or mobile proxies.
\ No newline at end of file
+The above logic is: requests are made with cheap proxies, such as datacenter proxies, until they are blocked, then retried with higher-quality proxies, such as residential or mobile proxies.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md b/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md
index bb0ab70..edc9d22 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md
@@ -1,7 +1,5 @@
# Requests & Responses
-This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
-
## The Request Object
A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:
@@ -95,9 +93,9 @@ yield response.follow("/page", referer_flow=False)
Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
-- **`dict`**: A scraped item, added to the results
-- **`Request`**: A follow-up request, added to the queue
-- **`None`**: Silently ignored
+- **`dict`** - A scraped item, added to the results
+- **`Request`** - A follow-up request, added to the queue
+- **`None`** - Silently ignored
```python
class MySpider(Spider):
@@ -116,7 +114,9 @@ class MySpider(Spider):
yield {"content": response.css("article::text").get("")}
```
-**Note:** All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.
+**Note:**
+
+All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.
## Request Priority
@@ -193,4 +193,4 @@ async def parse_product(self, response: Response):
When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence).
-The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.
\ No newline at end of file
+The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.
diff --git a/agent-skill/Scrapling-Skill/references/spiders/sessions.md b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
index e9b793a..277e13e 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/sessions.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/sessions.md
@@ -17,7 +17,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md
## Configuring Sessions
-Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance - use `manager.add()` to register sessions:
+Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance. Use `manager.add()` to register sessions:
```python
from scrapling.spiders import Spider, Response
@@ -46,7 +46,7 @@ The `manager.add()` method takes:
**Notes:**
1. In all requests, if you don't specify which session to use, the default session is used. The default session is determined in one of two ways:
- 1. The first session you add to the manager becomes the default automatically.
+ 1. The first session you add to the managed becomes the default automatically.
2. The session that gets `default=True` while added to the manager.
2. The instances you pass of each session don't have to be already started by you; the spider checks on all sessions if they are not already started and starts them.
3. If you want a specific session to start when used only, then use the `lazy` argument while adding that session to the manager. Example: start the browser only when you need it, not with the spider start.
@@ -100,7 +100,7 @@ The example keeps the browser instance and reads its retained API history once i
The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
-Sessions can also be different instances of the same class with different configurations:
+Note that the sessions don't have to be from different classes only, but can be the same session, but different instances with different configurations, for example, like below:
```python
from scrapling.spiders import Spider, Response
@@ -132,6 +132,8 @@ class ProductSpider(Spider):
}
```
+Or you can separate concerns and keep a session with its cookies/state for specific requests, etc...
+
## Session Arguments
Extra keyword arguments passed to a `Request` (or through `response.follow(**kwargs)`) are forwarded to the session's fetch method. This lets you customize individual requests without changing the session configuration:
@@ -155,9 +157,11 @@ async def parse(self, response: Response):
)
```
-**Warning:** When using `FetcherSession` in spiders, you cannot use `.get()` and `.post()` methods directly. By default, the request is an HTTP GET request; to use another HTTP method, pass it to the `method` argument as in the above example. This unifies the `Request` interface across all session types.
+**Warning:**
+
+Normally, when you use `FetcherSession`, `Fetcher`, or `AsyncFetcher`, you specify the HTTP method to use with the corresponding method like `.get()` and `.post()`. But while using `FetcherSession` in spiders, you can't do this. By default, the request is an _HTTP GET_ request; if you want to use another HTTP method, you have to pass it to the `method` argument, as in the above example. The reason for this is to unify the `Request` interface across all session types.
-For browser sessions (`AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
+With the `"stealth"` session (`AsyncStealthySession`) registered in the multi-session example above, you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
```python
async def parse(self, response: Response):
@@ -174,14 +178,16 @@ async def parse(self, response: Response):
yield response.follow(
"/dynamic-page",
- sid="browser",
+ sid="stealth",
callback=self.parse_dynamic,
wait_selector="div.loaded",
network_idle=True,
)
```
-**Warning:** Session arguments (**kwargs) passed from the original request are inherited by `response.follow()`, except `method`, `params`, `data`, `json`, `files`, and `multipart`, so following a link after a POST issues a plain GET. New kwargs take precedence over inherited ones.
+**Warning:**
+
+Session arguments (**kwargs) passed from the original request are inherited by `response.follow()`, except `method`, `params`, `data`, `json`, `files`, and `multipart`, so following a link after a POST issues a plain GET. New kwargs take precedence over inherited ones.
```python
from scrapling.spiders import Spider, Response
@@ -202,7 +208,7 @@ class ProductSpider(Spider):
next_page = response.css("a.next::attr(href)").get()
if next_page:
- yield Request(next_page)
+ yield response.follow(next_page)
async def parse_product(self, response: Response):
yield {
@@ -210,4 +216,6 @@ class ProductSpider(Spider):
"price": response.css(".price::text").get(""),
}
```
-**Note:** Upon spider closure, the manager automatically checks whether any sessions are still running and closes them before closing the spider.
\ No newline at end of file
+**Info:**
+
+No need to mention that, upon spider closure, the manager automatically checks whether any sessions are still running and closes them before closing the spider.