Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: update all pages to be clearer, fix mistakes, and improve for SEO

Karim shoairc8568a4149ae
diff --git a/README.md b/README.md
index 2135a44..5bcd8e4 100644
--- a/README.md
+++ b/README.md
@@ -69,10 +69,10 @@ Use it from Python, the terminal, or your AI agent through its MCP server and Ag
## What you can do with Scrapling
-- **Fetch & interact**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
-- **Parse & extract**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
-- **Crawl & export**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
-- **Scrape with AI or from your terminal**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
+- **[Fetch & interact](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
+- **[Parse & extract](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
+- **[Crawl & export](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
+- **[Scrape with AI or from your terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
Start with a single page:
@@ -84,22 +84,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Or scale up to full crawls:
+Or scale up with a spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+This prints `10` and saves the quotes and authors to `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -241,13 +248,13 @@ MySpider().start()
- 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.
- ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays.
- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
-- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
+- 💾 **Pause & Resume**: Save pending requests and seen URLs with `crawldir`. Press Ctrl+C to stop gracefully and restart with the same directory.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
- 🚦 **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
-- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.
+- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to collect products from public Shopify stores, one item per variant.
- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.
@@ -269,7 +276,7 @@ MySpider().start()
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
### AI Features
-- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA is faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -316,7 +323,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Keep t
data = page.css('#padded_content a').getall()
# Or use one-off request style, it opens the browser for this request, then closes it after finishing
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Full browser automation
@@ -335,7 +342,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
Build full crawlers with concurrent requests, multiple session types, and pause/resume:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -382,7 +389,7 @@ Pause and resume long crawls with checkpoints by running the spider like this:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.
+With `crawldir` set, press Ctrl+C to save pending requests and seen URLs. Restart with the same `crawldir` to resume. Save scraped items separately.
Or skip writing the crawling logic altogether with the ready-made templates, like pulling an entire Shopify store's catalog:
```python
@@ -393,6 +400,7 @@ class MyStore(ShopifySpider):
result = MyStore().start() # Every product in the store, one item per variant
```
+The template reads products exposed through public collection endpoints, one item per variant.
### Advanced Parsing & Navigation
```python
@@ -410,18 +418,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # and so on...
# Find element by text content
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Advanced navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Chained selectors
+quote_texts = page.css('.quote').css('.text::text').getall() # Chained selectors
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Element relationships and similarity
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
You can use the parser right away if you don't want to fetch websites like below:
```python
@@ -436,22 +445,18 @@ And it works precisely the same way!
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` is context-aware and can work in both sync/async patterns
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
-# Async session usage
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
-
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
-
- print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
+
+asyncio.run(main())
```
## CLI & Interactive Shell
@@ -462,13 +467,14 @@ Scrapling includes a powerful command-line interface:
Launch the interactive Web Scraping shell
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Extract pages to a file directly without programming (Extracts the content inside the `body` tag by default). If the output file ends with `.txt`, then the text content of the target will be extracted. If it ends in `.md`, it will be a Markdown representation of the HTML content; if it ends in `.html`, it will be the HTML content itself.
+Extract pages to a file directly without programming. Use `--css-selector` to pick the content, or `--ai-targeted` to select and sanitize the `body` content. Use `.txt` for text, `.md` for Markdown, or `.html` for HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -489,7 +495,7 @@ Scrapling isn't just powerful-it's also blazing fast. The following benchmarks c
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -499,8 +505,8 @@ Scrapling's adaptive element finding capabilities significantly outperform alter
| Library | Time (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
@@ -553,7 +559,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Remember that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
+ Run `scrapling install` when you need browser fetchers. HTTP requests do not need a browser installation.
### Docker
You can also install a Docker image with all extras and browsers with the following command from DockerHub:
@@ -583,7 +589,7 @@ If you have used our library for research purposes please quote us with the foll
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_AR.md b/docs/README_AR.md
index ecd92bb..28544b9 100644
--- a/docs/README_AR.md
+++ b/docs/README_AR.md
@@ -57,10 +57,10 @@ Scrapling هو إطار عمل لـ Web Scraping بلغة Python يجمع طلب
## ماذا يمكنك أن تفعل باستخدام Scrapling
-- **الجلب والتفاعل**: طلبات HTTP سريعة، ومتصفحات خفية، وتشغيل JavaScript، والتعامل مع Cloudflare Turnstile/Interstitial، وجلسات مستمرة، وتسجيل طلبات API في الخلفية.
-- **التحليل والاستخراج**: CSS وXPath، والبحث بالنص والتعبيرات النمطية، وتتبع العناصر التكيفي، ومطابقة العناصر المتشابهة، وإخراج Markdown نظيف.
-- **الزحف والتصدير**: عناكب متزامنة، وتدوير للبروكسي، وإعادة محاولات، وAutoThrottle، وإيقاف مؤقت واستئناف، وStreaming، وقوالب جاهزة، وتصدير بصيغ JSON/JSONL/CSV/XML.
-- **استخراج البيانات بالذكاء الاصطناعي أو من الطرفية**: خادم MCP لوكلاء الذكاء الاصطناعي لتصفح المواقع واستخراج البيانات، وAgent Skill، وأوامر CLI، وShell تفاعلي.
+- **[الجلب والتفاعل](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: طلبات HTTP سريعة، ومتصفحات خفية، وتشغيل JavaScript، والتعامل مع Cloudflare Turnstile/Interstitial، وجلسات مستمرة، وتسجيل طلبات API في الخلفية.
+- **[التحليل والاستخراج](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS وXPath، والبحث بالنص والتعبيرات النمطية، وتتبع العناصر التكيفي، ومطابقة العناصر المتشابهة، وإخراج Markdown نظيف.
+- **[الزحف والتصدير](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: عناكب متزامنة، وتدوير للبروكسي، وإعادة محاولات، وAutoThrottle، وإيقاف مؤقت واستئناف، وStreaming، وقوالب جاهزة، وتصدير بصيغ JSON/JSONL/CSV/XML.
+- **[استخراج البيانات بالذكاء الاصطناعي أو من الطرفية](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: خادم MCP لوكلاء الذكاء الاصطناعي لتصفح المواقع واستخراج البيانات، وAgent Skill، وأوامر CLI، وShell تفاعلي.
ابدأ بصفحة واحدة:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-أو توسع إلى عمليات زحف كاملة:
+أو توسع باستخدام Spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+يطبع `10` ويحفظ الاقتباسات وأسماء المؤلفين في `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **واجهة Spider شبيهة بـ Scrapy**: عرّف Spiders مع `start_urls`، و async `parse` callbacks، وكائنات `Request`/`Response`.
- ⚡ **زحف متزامن**: حدود تزامن قابلة للتكوين، وتحكم بالسرعة حسب النطاق، وتأخيرات التنزيل.
- 🔄 **دعم الجلسات المتعددة**: واجهة موحدة لطلبات HTTP، ومتصفحات خفية بدون واجهة في Spider واحد - وجّه الطلبات إلى جلسات مختلفة بالمعرّف.
-- 💾 **إيقاف واستئناف**: استمرارية الزحف القائمة على Checkpoint. اضغط Ctrl+C للإيقاف بسلاسة؛ أعد التشغيل للاستئناف من حيث توقفت.
+- 💾 **إيقاف واستئناف**: احفظ الطلبات المعلقة والروابط التي سبق زيارتها باستخدام `crawldir`. اضغط Ctrl+C للإيقاف بسلاسة، ثم أعد التشغيل بنفس المجلد.
- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية - مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة.
- 🛡️ **كشف الطلبات المحظورة**: كشف تلقائي وإعادة محاولة للطلبات المحظورة مع منطق قابل للتخصيص.
- 🚦 **AutoThrottle**: لا داعي لتخمين التأخير. يضبط Spider تأخير كل نطاق تلقائياً حسب سرعة استجابة الموقع، ثم يضاعفه (أو ينتظر ما يطلبه `Retry-After`) عندما يبدأ الموقع في حظرك أو تحديد معدل طلباتك، ويعود للتسريع عندما يتوقف عن ذلك.
- 🤖 **الامتثال لـ robots.txt**: خيار `robots_txt_obey` الاختياري الذي يحترم توجيهات `Disallow` و `Crawl-delay` و `Request-rate` مع التخزين المؤقت لكل نطاق.
- 🧪 **وضع التطوير**: تخزين الاستجابات على القرص في التشغيل الأول وإعادة تشغيلها في التشغيلات اللاحقة - كرّر العمل على منطق `parse()` دون الحاجة لإرسال طلبات جديدة إلى الخوادم المستهدفة.
-- 🧩 **قوالب Spider جاهزة**: تخطَّ الكود المتكرر مع `CrawlSpider` لتتبع الروابط بقواعد، و`SitemapSpider` للزحف المعتمد على sitemap/robots.txt، و`XMLFeedSpider`/`CSVFeedSpider` للمرور على خلاصات XML/RSS و CSV، و`ShopifySpider` لسحب كل منتج من أي متجر Shopify عبر واجهة JSON الخاصة به، مع عنصر لكل variant.
+- 🧩 **قوالب Spider جاهزة**: تخطَّ الكود المتكرر مع `CrawlSpider` لتتبع الروابط بقواعد، و`SitemapSpider` للزحف المعتمد على sitemap/robots.txt، و`XMLFeedSpider`/`CSVFeedSpider` للمرور على خلاصات XML/RSS و CSV، و`ShopifySpider` لجمع المنتجات من متاجر Shopify العامة، مع عنصر لكل variant.
- 🔗 **استخراج الروابط**: أداة `LinkExtractor` مستقلة مع أنماط allow/deny، وفلاتر النطاقات، وتحديد النطاق عبر CSS/XPath، وتصفية الامتدادات، وتوحيد الروابط - استخدمها داخل القوالب أو بمفردها.
- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو المصدّرات المدمجة JSON/JSONL/CSV/XML عبر `result.items.to_json()` و`to_jsonl()` و`to_csv()` و`to_xml()`.
### جلب متقدم للمواقع مع دعم الجلسات
- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP/3.
- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `StealthyFetcher` التي تدعم Chromium من Playwright و Google Chrome.
-- **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` وانتحال fingerprint. يمكنه تجاوز جميع أنواع Turnstile/Interstitial من Cloudflare بسهولة بالأتمتة.
+- **تجاوز مكافحة الروبوتات**: قدرات تخفٍ متقدمة مع `StealthyFetcher` وتمويه بصمة المتصفح. يمكنه تجاوز جميع أنواع Cloudflare Turnstile/Interstitial بسهولة وبشكل آلي.
- **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات.
- **تدوير Proxy**: `ProxyRotator` مدمج مع استراتيجيات التدوير الدوري أو المخصصة عبر جميع أنواع الجلسات، بالإضافة إلى تجاوزات Proxy لكل طلب.
- **حظر النطاقات والإعلانات**: حظر الطلبات إلى نطاقات محددة (ونطاقاتها الفرعية) أو تفعيل حظر الإعلانات المدمج (~3,500 نطاق إعلانات/تتبع معروف) في الجوالب المعتمدة على المتصفح.
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **البحث عن عناصر مشابهة**: تحديد العناصر المشابهة للعناصر الموجودة تلقائياً.
### ميزات الذكاء الاصطناعي
-- 🤖 **خادم MCP**: دع روبوتات الدردشة ووكلاء الذكاء الاصطناعي (Claude/Cursor/إلخ) يستخرجون البيانات عبر Scrapling بأدوات تعمل لمرة واحدة أو عبر جلسات، وتغطي طلبات HTTP العادية (بأي طريقة) وجلب الصفحات بالمتصفح الخفي الذي يتجاوز Cloudflare. يتم تضييق الصفحات بمحددات CSS وتنظيفها من محتوى حقن الأوامر قبل أن يراها الذكاء الاصطناعي، فيقرأ الوكيل أقل، ويكلف أقل، ولا يمكن اختطافه بنص مخفي. كما تتوفر لقطات الشاشة، والمتصفحات البعيدة عبر CDP، ونقل HTTP آمن افتراضياً. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **خادم MCP**: دع روبوتات المحادثة ووكلاء الذكاء الاصطناعي (Claude/Cursor وغيرها) يستخرجون البيانات عبر Scrapling باستخدام طلبات HTTP، والتفاعل مع المتصفح، ولقطات الشاشة، وحالة تسجيل الدخول المحفوظة، واستجابات API في الخلفية. يستخدم متصفحنا الخفي مع الأدوات المناسبة لتمكين نماذج الذكاء الاصطناعي المناسبة من تجاوز أي اختبار CAPTCHA تواجهه أثناء العمل (ناهيك عن أداة حل تحديات Cloudflare الخاصة بنا). يستخدم طرقاً متعددة لتقليل المحتوى المرسل للنموذج، وحظر الإعلانات، وحمايته من حقن التعليمات (Prompt-Injection). يشمل متصفحات بعيدة عبر CDP ونقل HTTP مع خيارات مصادقة. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: [مهارة جاهزة للتثبيت](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) تُعلّم وكلاء البرمجة المكتبة بالكامل، حتى يكون الكود الذي يكتبونه بـ Scrapling مطابقاً للواجهة الحالية بدلاً من التخمين.
- 📚 **Markdown جاهز لأنظمة RAG**: حوّل أي صفحة إلى Markdown نظيف ومعقّم وجاهز لنماذج اللغة بسطر واحد (`page.markdown()`)، أو ازحف موقعاً كاملاً إلى مجموعة ملفات Markdown باستخدام القالب `ContentSpider`، كل ذلك بدون استخدام أي نموذج لغوي في العملية. ([الوثائق](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # أبق
data = page.css('#padded_content a').getall()
# أو استخدم نمط الطلب لمرة واحدة، يفتح المتصفح لهذا الطلب، ثم يغلقه بعد الانتهاء
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
أتمتة المتصفح الكاملة
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
ابنِ زواحف كاملة مع طلبات متزامنة وأنواع جلسات متعددة وإيقاف/استئناف:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,17 +377,18 @@ class MultiSessionSpider(Spider):
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-اضغط Ctrl+C للإيقاف بسلاسة - يتم حفظ التقدم تلقائياً. لاحقاً، عند تشغيل Spider مرة أخرى، مرر نفس `crawldir`، وسيستأنف من حيث توقف.
+عند ضبط `crawldir`، اضغط Ctrl+C لحفظ الطلبات المعلقة والروابط التي سبق زيارتها. أعد التشغيل بنفس `crawldir` للاستئناف. احفظ البيانات المستخرجة بشكل منفصل.
-أو تخطَّ كتابة منطق الزحف تماماً باستخدام القوالب الجاهزة، مثل سحب كتالوج متجر Shopify بالكامل:
+أو تخطَّ كتابة منطق الزحف بالكامل باستخدام القوالب الجاهزة، مثل جلب كتالوج متجر Shopify بالكامل:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # كل منتج في المتجر، عنصر لكل variant
+result = MyStore().start() # كل منتج في المتجر، مع عنصر لكل variant
```
+يقرأ القالب المنتجات المتاحة عبر واجهات المجموعات العامة، مع عنصر لكل variant.
### التحليل المتقدم والتنقل
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # وهكذا...
# البحث عن عنصر بمحتوى النص
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# التنقل المتقدم
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # محددات متسلسلة
+quote_texts = page.css('.quote').css('.text::text').getall() # محددات متسلسلة
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# علاقات العناصر والتشابه
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
يمكنك استخدام المحلل مباشرة إذا كنت لا تريد جلب المواقع كما يلي:
```python
@@ -424,22 +433,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` واعٍ بالسياق ويعمل في كلا النمطين المتزامن/async
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# استخدام جلسة async
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # تتكيف `FetcherSession` مع السياق وتعمل بالنمطين المتزامن وغير المتزامن
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # اختياري - حالة مجموعة علامات تبويب المتصفح (مشغول/حر/خطأ)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## واجهة سطر الأوامر والـ Shell التفاعلي
@@ -450,13 +455,14 @@ async with AsyncStealthySession(max_pages=2) as session:
تشغيل Shell الـ Web Scraping التفاعلي
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-استخرج الصفحات إلى ملف مباشرة دون برمجة (يستخرج المحتوى داخل وسم `body` افتراضياً). إذا انتهى ملف الإخراج بـ `.txt`، فسيتم استخراج محتوى النص للهدف. إذا انتهى بـ `.md`، فسيكون تمثيل Markdown لمحتوى HTML؛ إذا انتهى بـ `.html`، فسيكون محتوى HTML نفسه.
+استخرج الصفحات إلى ملف مباشرة دون برمجة. استخدم `--css-selector` لاختيار المحتوى، أو `--ai-targeted` لاختيار محتوى `body` وتنقيته. استخدم `.txt` للنص، أو `.md` لصيغة Markdown، أو `.html` لصيغة HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # جميع العناصر المطابقة لمحدد CSS '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- تذكر أنك تحتاج إلى تثبيت تبعيات المتصفح مع `scrapling install` بعد أي من هذه الإضافات (إذا لم تكن قد فعلت ذلك بالفعل)
+ شغّل `scrapling install` عند استخدام جلب المتصفح. طلبات HTTP لا تحتاج إلى تثبيت متصفح.
### Docker
يمكنك أيضاً تثبيت صورة Docker مع جميع الإضافات والمتصفحات باستخدام الأمر التالي من DockerHub:
@@ -571,7 +577,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_CN.md b/docs/README_CN.md
index 1b0380a..4a22467 100644
--- a/docs/README_CN.md
+++ b/docs/README_CN.md
@@ -57,10 +57,10 @@ Scrapling 是一个 Python 网页抓取框架,将快速 HTTP 请求、隐身
## 用 Scrapling 能做什么
-- **获取与交互**:快速 HTTP 请求、隐身浏览器、JavaScript 渲染、Cloudflare Turnstile/Interstitial 处理、持久会话,以及后台 API 通信捕获。
-- **解析与提取**:CSS、XPath、文本和正则表达式搜索、自适应元素跟踪、相似元素匹配,以及干净的 Markdown 输出。
-- **爬取与导出**:并发爬虫、代理轮换、重试、AutoThrottle、暂停/恢复、流式输出、现成的爬虫模板,以及 JSON/JSONL/CSV/XML 导出。
-- **使用 AI 或终端抓取**:供 AI 智能体浏览网页和提取数据的 MCP 服务器、Agent Skill、CLI 命令和交互式 Shell。
+- **[获取与交互](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**:快速 HTTP 请求、隐身浏览器、JavaScript 渲染、Cloudflare Turnstile/Interstitial 处理、持久会话,以及后台 API 通信捕获。
+- **[解析与提取](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**:CSS、XPath、文本和正则表达式搜索、自适应元素跟踪、相似元素匹配,以及干净的 Markdown 输出。
+- **[爬取与导出](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**:并发爬虫、代理轮换、重试、AutoThrottle、暂停/恢复、流式输出、现成的爬虫模板,以及 JSON/JSONL/CSV/XML 导出。
+- **[使用 AI 或终端抓取](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**:供 AI 智能体浏览网页和提取数据的 MCP 服务器、Agent Skill、CLI 命令和交互式 Shell。
从单个页面开始:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-或扩展为完整爬取:
+也可以用 Spider 扩大抓取规模:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+输出 `10`,并将名言和作者保存到 `quotes.json`。
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **类 Scrapy 的 Spider API**:使用 `start_urls`、async `parse` callback 和`Request`/`Response` 对象定义 Spider。
- ⚡ **并发爬取**:可配置的并发限制、按域名节流和下载延迟。
- 🔄 **多 Session 支持**:统一接口,支持 HTTP 请求和隐秘无头浏览器在同一个 Spider 中使用--通过 ID 将请求路由到不同的 Session。
-- 💾 **暂停与恢复**:基于 Checkpoint 的爬取持久化。按 Ctrl+C 优雅关闭;重启后从上次停止的地方继续。
+- 💾 **暂停与恢复**:通过 `crawldir` 保存待处理请求和已访问 URL。按 Ctrl+C 正常停止,再使用同一目录重启。
- 📡 **Streaming 模式**:通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据--非常适合 UI、管道和长时间运行的爬取。
- 🛡️ **被阻止请求检测**:自动检测并重试被阻止的请求,支持自定义逻辑。
- 🚦 **AutoThrottle 自动限速**:不用再猜延迟。Spider 会根据网站的响应速度自动调整每个域名的延迟,当网站开始封禁或限流时把延迟翻倍(或按 `Retry-After` 要求等待),并在恢复正常后重新提速。
- 🤖 **robots.txt 合规**:可选的 `robots_txt_obey` 标志,支持 `Disallow`、`Crawl-delay` 和 `Request-rate` 指令,并按域名缓存。
- 🧪 **开发模式**:首次运行时将响应缓存到磁盘,后续运行时直接回放 - 在不重新请求目标服务器的情况下迭代你的 `parse()` 逻辑。
-- 🧩 **现成的 Spider 模板**:跳过样板代码--使用 `CrawlSpider` 基于规则跟踪链接,`SitemapSpider` 基于 sitemap/robots.txt 爬取,`XMLFeedSpider`/`CSVFeedSpider` 用于遍历 XML/RSS 和 CSV 数据源,以及 `ShopifySpider` 通过 JSON API 抓取任意 Shopify 商店的全部商品,每个变体一条数据。
+- 🧩 **现成的 Spider 模板**:跳过样板代码--使用 `CrawlSpider` 基于规则跟踪链接,`SitemapSpider` 基于 sitemap/robots.txt 爬取,`XMLFeedSpider`/`CSVFeedSpider` 用于遍历 XML/RSS 和 CSV 数据源,以及 `ShopifySpider` 用于收集公开 Shopify 店铺中的商品,每个变体一条记录。
- 🔗 **链接提取**:独立的 `LinkExtractor` 组件,支持 allow/deny 模式、域名过滤、CSS/XPath 范围限定、扩展名过滤和链接规范化--可在模板中使用,也可单独使用。
- 📦 **内置导出**:通过钩子和您自己的管道导出结果,或使用内置的 JSON/JSONL/CSV/XML 导出器:`result.items.to_json()`、`to_jsonl()`、`to_csv()` 和 `to_xml()`。
### 支持 Session 的高级网站获取
- **HTTP 请求**:使用 `Fetcher` 类进行快速和隐秘的 HTTP 请求。可以模拟浏览器的 TLS fingerprint、标头并使用 HTTP/3。
- **动态加载**:通过 `StealthyFetcher` 类使用完整的浏览器自动化获取动态网站,支持 Playwright 的 Chromium 和 Google Chrome。
-- **反机器人绕过**:使用 `StealthyFetcher` 的高级隐秘功能和 fingerprint 伪装。可以轻松自动绕过所有类型的 Cloudflare Turnstile/Interstitial。
+- **反机器人绕过**:`StealthyFetcher` 提供高级隐身能力和指纹伪装,可通过自动化轻松绕过所有类型的 Cloudflare Turnstile/Interstitial。
- **Session 管理**:使用 `FetcherSession` 和 `StealthySession` 类实现持久化 Session 支持,用于跨请求的 cookie 和状态管理。
- **Proxy 轮换**:内置 `ProxyRotator`,支持轮询或自定义策略,适用于所有 Session 类型,并支持按请求覆盖 Proxy。
- **域名和广告屏蔽**:在基于浏览器的 Fetcher 中屏蔽对特定域名(及其子域名)的请求,或启用内置广告屏蔽(约 3,500 个已知广告/追踪域名)。
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **查找相似元素**:自动定位与已找到元素相似的元素。
### AI 功能
-- 🤖 **MCP 服务器**:让 AI 聊天机器人和智能体(Claude/Cursor 等)通过 Scrapling 抓取网页,提供一次性工具和基于会话的工具,覆盖普通 HTTP 请求(任意方法)以及可绕过 Cloudflare 的隐身浏览器抓取。页面在交给 AI 之前会先用 CSS 选择器缩小范围并清除提示注入内容,因此智能体读得更少、花得更少,也不会被隐藏文本劫持。还包含页面截图、通过 CDP 控制远程浏览器,以及默认安全的 HTTP 传输。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP 服务器**:让 AI 聊天机器人和智能体(Claude/Cursor 等)通过 Scrapling 抓取网页,使用 HTTP 请求、浏览器操作、截图、已保存的登录状态和后台 API 响应。借助我们的隐身浏览器和合适的工具,让具备相应能力的 AI 模型能够绕过工作中遇到的任何 CAPTCHA(更不用说我们自带的 Cloudflare 自动解题功能)。通过多种方法减少发送给模型的内容、屏蔽广告,并保护模型免受提示注入。还支持通过 CDP 连接远程浏览器,以及带身份验证选项的 HTTP 传输。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:[开箱即用的 Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html),让编码智能体全面掌握本库,使它们用 Scrapling 写出的代码符合当前 API,而不是靠猜测。
- 📚 **RAG 就绪的 Markdown**:用一行代码(`page.markdown()`)把任意页面转换为干净、经过净化、可直接供 LLM 使用的 Markdown,或使用 `ContentSpider` 模板把整个网站抓取为 Markdown 语料库,整个过程无需调用 LLM。([文档](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # 保持
data = page.css('#padded_content a').getall()
# 或使用一次性请求样式,为此请求打开浏览器,完成后关闭
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
完整的浏览器自动化
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spider
构建具有并发请求、多种 Session 类型和暂停/恢复功能的完整爬虫:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,17 +377,18 @@ class MultiSessionSpider(Spider):
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-按 Ctrl+C 优雅暂停--进度会自动保存。之后,当您再次启动 Spider 时,传递相同的 `crawldir`,它将从上次停止的地方继续。
+设置 `crawldir` 后,按 Ctrl+C 保存待处理请求和已访问 URL。使用同一个 `crawldir` 重启即可继续。请单独保存提取的数据。
-或者使用现成的模板,完全跳过爬取逻辑的编写,例如抓取整个 Shopify 商店的商品目录:
+也可以用现成模板直接省去编写爬取逻辑,比如获取整个 Shopify 店铺的商品目录:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # 商店中的每件商品,每个变体一条数据
+result = MyStore().start() # 店铺中的所有商品,每个变体一条记录
```
+模板读取公开集合接口中提供的商品,每个变体一条记录。
### 高级解析与导航
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # 等等...
# 按文本内容查找元素
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# 高级导航
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # 链式选择器
+quote_texts = page.css('.quote').css('.text::text').getall() # 链式选择器
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# 元素关系和相似性
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
如果您不想获取网站,可以直接使用解析器,如下所示:
```python
@@ -424,22 +433,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession`是上下文感知的,可以在 sync/async 模式下工作
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# Async Session 用法
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` 能识别上下文,同时支持同步和异步用法
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # 可选 - 浏览器标签池的状态(忙/空闲/错误)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI 和交互式 Shell
@@ -450,13 +455,14 @@ Scrapling 包含强大的命令行界面:
启动交互式 Web Scraping Shell
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-直接将页面提取到文件而无需编程(默认提取 `body` 标签内的内容)。如果输出文件以`.txt` 结尾,则将提取目标的文本内容。如果以`.md` 结尾,它将是 HTML 内容的 Markdown 表示;如果以`.html` 结尾,它将是 HTML 内容本身。
+无需编程即可将页面直接提取到文件。使用 `--css-selector` 选择内容,或用 `--ai-targeted` 选择并清理 `body` 内容。使用 `.txt` 输出文本、`.md` 输出 Markdown,或 `.html` 输出 HTML。
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # 所有匹配 CSS 选择器'#fromSkipToProducts' 的元素
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -477,7 +483,7 @@ Scrapling 不仅功能强大--它还速度极快。以下基准测试将 Scrapli
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -487,8 +493,8 @@ Scrapling 的自适应元素查找功能明显优于替代方案:
| 库 | 时间 (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> 所有基准测试代表 100+ 次运行的平均值。请参阅 [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) 了解方法。
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- 请记住,在安装任何这些额外功能后(如果您还没有安装),您需要使用 `scrapling install` 安装浏览器依赖项
+ 需要浏览器抓取时再运行 `scrapling install`。HTTP 请求无需安装浏览器。
### Docker
您还可以使用以下命令从 DockerHub 安装包含所有额外功能和浏览器的 Docker 镜像:
@@ -571,7 +577,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_DE.md b/docs/README_DE.md
index d1cd9bd..7b1aece 100644
--- a/docs/README_DE.md
+++ b/docs/README_DE.md
@@ -57,10 +57,10 @@ Nutzen Sie Scrapling in Python, im Terminal oder mit Ihrem KI-Agenten über den
## Was Sie mit Scrapling machen können
-- **Abrufen & interagieren**: Schnelle HTTP-Anfragen, Stealth-Browser, JavaScript-Rendering, Umgang mit Cloudflare Turnstile/Interstitial, persistente Sessions und Aufzeichnung von API-Aufrufen im Hintergrund.
-- **Parsen & extrahieren**: CSS, XPath, Text- und Regex-Suche, adaptives Element-Tracking, Suche nach ähnlichen Elementen und saubere Markdown-Ausgabe.
-- **Crawlen & exportieren**: Parallele Spiders, Proxy-Rotation, Wiederholungsversuche, AutoThrottle, Pause & Resume, Streaming, fertige Vorlagen und JSON/JSONL/CSV/XML-Exporte.
-- **Scrapen mit KI oder im Terminal**: Ein MCP-Server für KI-Agenten zum Browsen und Extrahieren von Daten, ein Agent Skill, CLI-Befehle und eine interaktive Shell.
+- **[Abrufen & interagieren](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Schnelle HTTP-Anfragen, Stealth-Browser, JavaScript-Rendering, Umgang mit Cloudflare Turnstile/Interstitial, persistente Sessions und Aufzeichnung von API-Aufrufen im Hintergrund.
+- **[Parsen & extrahieren](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, Text- und Regex-Suche, adaptives Element-Tracking, Suche nach ähnlichen Elementen und saubere Markdown-Ausgabe.
+- **[Crawlen & exportieren](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Parallele Spiders, Proxy-Rotation, Wiederholungsversuche, AutoThrottle, Pause & Resume, Streaming, fertige Vorlagen und JSON/JSONL/CSV/XML-Exporte.
+- **[Scrapen mit KI oder im Terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: Ein MCP-Server für KI-Agenten zum Browsen und Extrahieren von Daten, ein Agent Skill, CLI-Befehle und eine interaktive Shell.
Starten Sie mit einer einzelnen Seite:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Oder auf vollständige Crawls hochskalieren:
+Oder skalieren Sie mit einem Spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+Das gibt `10` aus und speichert Zitate und Autoren in `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **Scrapy-ähnliche Spider-API**: Definieren Sie Spiders mit `start_urls`, async `parse` Callbacks und `Request`/`Response`-Objekten.
- ⚡ **Paralleles Crawling**: Konfigurierbare Parallelitätslimits, domainbezogenes Throttling und Download-Verzögerungen.
- 🔄 **Multi-Session-Unterstützung**: Einheitliche Schnittstelle für HTTP-Anfragen und heimliche Headless-Browser in einem einzigen Spider -- leiten Sie Anfragen per ID an verschiedene Sessions weiter.
-- 💾 **Pause & Resume**: Checkpoint-basierte Crawl-Persistenz. Drücken Sie Strg+C für ein kontrolliertes Herunterfahren; starten Sie neu, um dort fortzufahren, wo Sie aufgehört haben.
+- 💾 **Pause & Resume**: Ausstehende Anfragen und besuchte URLs mit `crawldir` speichern. Mit Strg+C kontrolliert stoppen und mit demselben Verzeichnis neu starten.
- 📡 **Streaming-Modus**: Gescrapte Elemente in Echtzeit streamen über `async for item in spider.stream()` mit Echtzeit-Statistiken -- ideal für UI, Pipelines und lang laufende Crawls.
- 🛡️ **Erkennung blockierter Anfragen**: Automatische Erkennung und Wiederholung blockierter Anfragen mit anpassbarer Logik.
- 🚦 **AutoThrottle**: Schluss mit dem Raten von Verzögerungen. Der Spider stimmt die Verzögerung jeder Domain selbst darauf ab, wie schnell die Website antwortet, verdoppelt sie (oder wartet, was `Retry-After` verlangt), sobald die Website Sie blockiert oder ausbremst, und wird wieder schneller, sobald das aufhört.
- 🤖 **robots.txt-Konformität**: Optionales `robots_txt_obey`-Flag, das `Disallow`-, `Crawl-delay`- und `Request-rate`-Direktiven mit domainbasiertem Caching respektiert.
- 🧪 **Entwicklungsmodus**: Antworten beim ersten Lauf auf der Festplatte zwischenspeichern und bei weiteren Läufen erneut abspielen - iterieren Sie an Ihrer `parse()`-Logik, ohne die Zielserver erneut abzufragen.
-- 🧩 **Fertige Spider-Vorlagen**: Sparen Sie sich den Boilerplate-Code mit `CrawlSpider` für regelbasiertes Link-Following, `SitemapSpider` für Sitemap-/robots.txt-gesteuerte Crawls und `ShopifySpider`, um über die JSON-API jedes Produkt aus einem beliebigen Shopify-Shop zu ziehen -- ein Item pro Variante.
+- 🧩 **Fertige Spider-Vorlagen**: Sparen Sie sich den Boilerplate-Code mit `CrawlSpider` für regelbasiertes Link-Following, `SitemapSpider` für Sitemap-/robots.txt-gesteuerte Crawls, `XMLFeedSpider`/`CSVFeedSpider` für XML/RSS- und CSV-Feeds und `ShopifySpider` zum Sammeln von Produkten aus öffentlichen Shopify-Shops, ein Item pro Variante.
- 🔗 **Link-Extraktion**: Ein eigenständiges `LinkExtractor`-Primitiv mit Allow-/Deny-Mustern, Domain-Filtern, CSS-/XPath-Eingrenzung, Erweiterungsfilterung und Kanonisierung -- nutzbar innerhalb der Vorlagen oder für sich allein.
- 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder die integrierten JSON/JSONL/CSV/XML-Exporte mit `result.items.to_json()`, `to_jsonl()`, `to_csv()` und `to_xml()` exportieren.
### Erweitertes Website-Abrufen mit Session-Unterstützung
- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerprints und Header imitieren und HTTP/3 verwenden.
- **Dynamisches Laden**: Dynamische Websites mit vollständiger Browser-Automatisierung über die `StealthyFetcher`-Klasse abrufen, die Playwrights Chromium und Google Chrome unterstützt.
-- **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` und Fingerprint-Spoofing. Kann alle Arten von Cloudflares Turnstile/Interstitial einfach mit Automatisierung umgehen.
+- **Anti-Bot-Umgehung**: Fortschrittliche Stealth-Funktionen mit `StealthyFetcher` und Fingerprint-Spoofing. Umgeht mit Automatisierung mühelos alle Arten von Cloudflare-Turnstile-/Interstitial-Challenges.
- **Session-Verwaltung**: Persistente Session-Unterstützung mit den Klassen `FetcherSession` und `StealthySession` für Cookie- und Zustandsverwaltung über Anfragen hinweg.
- **Proxy-Rotation**: Integrierter `ProxyRotator` mit zyklischen oder benutzerdefinierten Rotationsstrategien über alle Session-Typen hinweg, plus Proxy-Überschreibungen pro Anfrage.
- **Domain- & Werbeblockierung**: Anfragen an bestimmte Domains (und deren Subdomains) blockieren oder die integrierte Werbeblockierung (~3.500 bekannte Werbe-/Tracker-Domains) in browserbasierten Fetchern aktivieren.
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **Ähnliche Elemente finden**: Elemente, die gefundenen Elementen ähnlich sind, automatisch lokalisieren.
### KI-Funktionen
-- 🤖 **MCP-Server**: Lassen Sie KI-Chatbots und -Agenten (Claude/Cursor/etc.) über Scrapling scrapen, mit Einmal- oder sitzungsbasierten Tools für einfache HTTP-Anfragen (jede Methode) und Stealth-Browser-Abrufe, die Cloudflare umgehen. Seiten werden mit CSS-Selektoren eingegrenzt und von Prompt-Injection-Inhalten bereinigt, bevor die KI sie sieht, sodass der Agent weniger liest, weniger kostet und nicht durch versteckten Text gekapert werden kann. Screenshots, Remote-Browser über CDP und ein standardmäßig abgesicherter HTTP-Transport sind enthalten. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP-Server**: Lassen Sie KI-Chatbots und -Agenten (Claude/Cursor/usw.) über Scrapling scrapen, mit HTTP-Anfragen, Browser-Aktionen, Screenshots, gespeicherten Anmeldungen und API-Antworten im Hintergrund. Mit unserem Stealth-Browser und den passenden Tools können geeignete KI-Modelle jedes CAPTCHA umgehen, das ihnen bei der Arbeit begegnet (ganz zu schweigen von unserem Cloudflare-Löser). Verschiedene Methoden reduzieren den Inhalt für das Modell, blockieren Werbung und schützen es vor Prompt-Injection. Remote-Browser über CDP und HTTP-Transport mit Authentifizierungsoptionen sind enthalten. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Ein installationsfertiger [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), der Coding-Agents die gesamte Bibliothek beibringt, damit der Code, den sie mit Scrapling schreiben, zur aktuellen API passt, statt geraten zu werden.
- 📚 **RAG-fertiges Markdown**: Wandeln Sie jede Seite mit einer Zeile (`page.markdown()`) in sauberes, bereinigtes, LLM-fertiges Markdown um oder crawlen Sie eine ganze Website mit der Vorlage `ContentSpider` in einen Markdown-Korpus, ganz ohne LLM im Prozess. ([Dokumentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Browse
data = page.css('#padded_content a').getall()
# Oder einmaligen Anfragenstil verwenden: öffnet den Browser für diese Anfrage und schließt ihn nach Abschluss
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Vollständige Browser-Automatisierung
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
Vollständige Crawler mit parallelen Anfragen, mehreren Session-Typen und Pause & Resume erstellen:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,7 +377,7 @@ Lange Crawls mit Checkpoints pausieren und fortsetzen, indem Sie den Spider so s
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Drücken Sie Strg+C, um kontrolliert zu pausieren -- der Fortschritt wird automatisch gespeichert. Wenn Sie den Spider später erneut starten, übergeben Sie dasselbe `crawldir`, und er setzt dort fort, wo er aufgehört hat.
+Mit gesetztem `crawldir` speichert Strg+C ausstehende Anfragen und bereits besuchte URLs. Starten Sie mit demselben `crawldir` neu, um fortzufahren. Speichern Sie extrahierte Daten separat.
Oder überspringen Sie das Schreiben der Crawling-Logik komplett mit den fertigen Vorlagen, etwa um den gesamten Katalog eines Shopify-Shops abzurufen:
```python
@@ -379,8 +386,9 @@ from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # Jedes Produkt im Shop, ein Item pro Variante
+result = MyStore().start() # Alle Produkte im Shop, ein Item pro Variante
```
+Die Vorlage liest Produkte aus öffentlichen Collection-Endpunkten, ein Item pro Variante.
### Erweitertes Parsing & Navigation
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # und so weiter...
# Element nach Textinhalt finden
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Erweiterte Navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Verkettete Selektoren
+quote_texts = page.css('.quote').css('.text::text').getall() # Verkettete Selektoren
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Elementbeziehungen und Ähnlichkeit
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
Sie können den Parser direkt verwenden, wenn Sie keine Websites abrufen möchten, wie unten gezeigt:
```python
@@ -424,22 +433,18 @@ Und es funktioniert genau auf die gleiche Weise!
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` ist kontextbewusst und kann sowohl in sync- als auch in async-Mustern arbeiten
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# Async-Session-Verwendung
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` passt sich dem Kontext an und funktioniert synchron wie asynchron
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # Optional - Der Status des Browser-Tab-Pools (beschäftigt/frei/Fehler)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI & Interaktive Shell
@@ -450,13 +455,14 @@ Scrapling enthält eine leistungsstarke Befehlszeilenschnittstelle:
Interaktive Web-Scraping-Shell starten
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Seiten direkt ohne Programmierung in eine Datei extrahieren (extrahiert standardmäßig den Inhalt im `body`-Tag). Wenn die Ausgabedatei mit `.txt` endet, wird der Textinhalt des Ziels extrahiert. Wenn sie mit `.md` endet, ist es eine Markdown-Darstellung des HTML-Inhalts; wenn sie mit `.html` endet, ist es der HTML-Inhalt selbst.
+Seiten direkt ohne Programmierung in eine Datei extrahieren. Wählen Sie Inhalte mit `--css-selector` aus oder verwenden Sie `--ai-targeted`, um den `body`-Inhalt auszuwählen und zu bereinigen. Nutzen Sie `.txt` für Text, `.md` für Markdown oder `.html` für HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Alle Elemente, die dem CSS-Selektor '#fromSkipToProducts' entsprechen
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Denken Sie daran, dass Sie nach einem dieser Extras (falls noch nicht geschehen) die Browser-Abhängigkeiten mit `scrapling install` installieren müssen
+ Führen Sie `scrapling install` aus, wenn Sie Browser-Fetcher benötigen. HTTP-Anfragen brauchen keine Browser-Installation.
### Docker
Sie können auch ein Docker-Image mit allen Extras und Browsern mit dem folgenden Befehl von DockerHub installieren:
@@ -571,7 +577,7 @@ Wenn Sie unsere Bibliothek für Forschungszwecke verwendet haben, zitieren Sie u
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_ES.md b/docs/README_ES.md
index b001c57..fdb030e 100644
--- a/docs/README_ES.md
+++ b/docs/README_ES.md
@@ -57,10 +57,10 @@ Extrae datos de páginas estáticas, renderiza JavaScript, interactúa con sitio
## Qué puedes hacer con Scrapling
-- **Obtener páginas e interactuar**: Solicitudes HTTP rápidas, navegadores sigilosos, renderizado de JavaScript, manejo de Cloudflare Turnstile/Interstitial, sesiones persistentes y captura de llamadas API en segundo plano.
-- **Analizar y extraer**: CSS, XPath, búsqueda por texto y expresiones regulares, seguimiento adaptativo de elementos, búsqueda de elementos similares y salida en Markdown limpio.
-- **Rastrear y exportar**: Spiders concurrentes, rotación de proxies, reintentos, AutoThrottle, pausa/reanudación, streaming, plantillas listas para usar y exportación a JSON/JSONL/CSV/XML.
-- **Hacer scraping con IA o desde tu terminal**: Un servidor MCP para que los agentes de IA naveguen y extraigan datos, un Agent Skill, comandos CLI y una consola interactiva.
+- **[Obtener páginas e interactuar](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Solicitudes HTTP rápidas, navegadores sigilosos, renderizado de JavaScript, manejo de Cloudflare Turnstile/Interstitial, sesiones persistentes y captura de llamadas API en segundo plano.
+- **[Analizar y extraer](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, búsqueda por texto y expresiones regulares, seguimiento adaptativo de elementos, búsqueda de elementos similares y salida en Markdown limpio.
+- **[Rastrear y exportar](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Spiders concurrentes, rotación de proxies, reintentos, AutoThrottle, pausa/reanudación, streaming, plantillas listas para usar y exportación a JSON/JSONL/CSV/XML.
+- **[Hacer scraping con IA o desde tu terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: Un servidor MCP para que los agentes de IA naveguen y extraigan datos, un Agent Skill, comandos CLI y una consola interactiva.
Empieza con una sola página:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-O escala a rastreos completos:
+O escala con un Spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+Imprime `10` y guarda las citas y sus autores en `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **API de Spider al estilo Scrapy**: Define spiders con `start_urls`, callbacks async `parse`, y objetos `Request`/`Response`.
- ⚡ **Rastreo Concurrente**: Límites de concurrencia configurables, limitación por dominio y retrasos de descarga.
- 🔄 **Soporte Multi-Session**: Interfaz unificada para solicitudes HTTP y navegadores headless sigilosos en un solo Spider - enruta solicitudes a diferentes sesiones por ID.
-- 💾 **Pause & Resume**: Persistencia de rastreo basada en Checkpoint. Presiona Ctrl+C para un cierre ordenado; reinicia para continuar desde donde lo dejaste.
+- 💾 **Pausa y reanudación**: Guarda las solicitudes pendientes y las URL ya vistas con `crawldir`. Detén el proceso con Ctrl+C y reinicia con el mismo directorio.
- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real - ideal para UI, pipelines y rastreos de larga duración.
- 🛡️ **Detección de Solicitudes Bloqueadas**: Detección automática y reintento de solicitudes bloqueadas con lógica personalizable.
- 🚦 **AutoThrottle**: Deja de adivinar los retrasos. El Spider ajusta por sí mismo el retraso de cada dominio según lo rápido que responda el sitio, lo duplica (o espera lo que pida `Retry-After`) cuando el sitio empieza a bloquearte o a limitarte, y vuelve a acelerar cuando deja de hacerlo.
- 🤖 **Cumplimiento de robots.txt**: Flag opcional `robots_txt_obey` que respeta las directivas `Disallow`, `Crawl-delay` y `Request-rate` con caché por dominio.
- 🧪 **Modo de Desarrollo**: Almacena las respuestas en disco en la primera ejecución y las reproduce en ejecuciones posteriores - itera sobre tu lógica de `parse()` sin volver a consultar los servidores objetivo.
-- 🧩 **Plantillas de Spider Listas para Usar**: Olvídate del código repetitivo con `CrawlSpider` para seguir enlaces por reglas, `SitemapSpider` para rastreos guiados por sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` para iterar feeds XML/RSS y CSV, y `ShopifySpider` para extraer todos los productos de cualquier tienda Shopify a través de su API JSON, un ítem por variante.
+- 🧩 **Plantillas de Spider Listas para Usar**: Olvídate del código repetitivo con `CrawlSpider` para seguir enlaces por reglas, `SitemapSpider` para rastreos guiados por sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` para iterar feeds XML/RSS y CSV, y `ShopifySpider` para recopilar productos de tiendas públicas de Shopify, un elemento por variante.
- 🔗 **Extracción de Enlaces**: Un primitivo `LinkExtractor` independiente con patrones allow/deny, filtros de dominio, delimitación por CSS/XPath, filtrado de extensiones y canonicalización - úsalo dentro de las plantillas o por su cuenta.
- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o los exportadores integrados JSON/JSONL/CSV/XML con `result.items.to_json()`, `to_jsonl()`, `to_csv()` y `to_xml()`.
### Obtención Avanzada de Sitios Web con Soporte de Session
- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar el fingerprint TLS de los navegadores, encabezados y usar HTTP/3.
- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `StealthyFetcher` compatible con Chromium de Playwright y Google Chrome.
-- **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` y falsificación de fingerprint. Puede evadir fácilmente todos los tipos de Turnstile/Interstitial de Cloudflare con automatización.
+- **Evasión anti-bot**: Capacidades avanzadas de sigilo con `StealthyFetcher` y suplantación de huellas. Puede superar fácilmente todo tipo de Turnstile/Interstitial de Cloudflare con automatización.
- **Gestión de Session**: Soporte de sesión persistente con las clases `FetcherSession` y `StealthySession` para la gestión de cookies y estado entre solicitudes.
- **Rotación de Proxy**: `ProxyRotator` integrado con estrategias de rotación cíclica o personalizadas en todos los tipos de sesión, además de sobrescrituras de Proxy por solicitud.
- **Bloqueo de Dominios y Anuncios**: Bloquea solicitudes a dominios específicos (y sus subdominios) o activa el bloqueo de anuncios integrado (~3,500 dominios de anuncios/rastreadores conocidos) en fetchers basados en navegador.
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **Encontrar Elementos Similares**: Localiza automáticamente elementos similares a los elementos encontrados.
### Características de IA
-- 🤖 **Servidor MCP**: Permite que los chatbots y agentes de IA (Claude/Cursor/etc) hagan scraping a través de Scrapling con herramientas de un solo uso o basadas en sesiones que cubren peticiones HTTP simples (cualquier método) y cargas con navegador sigiloso que evitan Cloudflare. Las páginas se acotan con selectores CSS y se limpian de contenido de inyección de prompts antes de que la IA las vea, así que el agente lee menos, cuesta menos y no puede ser secuestrado por texto oculto. Incluye capturas de pantalla, navegadores remotos por CDP y un transporte HTTP seguro por defecto. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Servidor MCP**: Permite que los chatbots y agentes de IA (Claude/Cursor/etc.) hagan scraping a través de Scrapling con solicitudes HTTP, acciones del navegador, capturas de pantalla, sesiones de inicio guardadas y respuestas API en segundo plano. Usa nuestro navegador sigiloso con las herramientas adecuadas para que los modelos de IA adecuados superen cualquier CAPTCHA que encuentren mientras trabajan (sin mencionar nuestro solucionador de Cloudflare). Usa muchos métodos para reducir el contenido enviado al modelo, bloquear anuncios y protegerlo contra Prompt-Injection. Incluye navegadores remotos por CDP y transporte HTTP con opciones de autenticación. ([Vídeo de demostración](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) listo para instalar que enseña la biblioteca completa a los agentes de programación, para que el código que escriban con Scrapling coincida con la API actual en vez de adivinarla.
- 📚 **Markdown listo para RAG**: Convierte cualquier página en Markdown limpio, saneado y listo para LLM con una sola línea (`page.markdown()`), o rastrea un sitio web completo hacia un corpus de Markdown con la plantilla `ContentSpider`, todo sin un LLM en el proceso. ([documentación](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Manté
data = page.css('#padded_content a').getall()
# O usa el estilo de solicitud de una sola vez, abre el navegador para esta solicitud, luego lo cierra después de terminar
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Automatización completa del navegador
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
Construye rastreadores completos con solicitudes concurrentes, múltiples tipos de sesión y Pause & Resume:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,17 +377,18 @@ Pausa y reanuda rastreos largos con checkpoints ejecutando el Spider así:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Presiona Ctrl+C para pausar de forma ordenada - el progreso se guarda automáticamente. Después, cuando inicies el Spider de nuevo, pasa el mismo `crawldir`, y continuará desde donde se detuvo.
+Con `crawldir` configurado, pulsa Ctrl+C para guardar las solicitudes pendientes y las URL ya vistas. Reinicia con el mismo `crawldir` para continuar. Guarda los datos extraídos por separado.
-O sáltate por completo la lógica de rastreo con las plantillas listas para usar, como extraer el catálogo entero de una tienda Shopify:
+O sáltate por completo la lógica de rastreo con las plantillas listas para usar, como extraer el catálogo completo de una tienda Shopify:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # Todos los productos de la tienda, un ítem por variante
+result = MyStore().start() # Todos los productos de la tienda, un elemento por variante
```
+La plantilla lee los productos expuestos por los endpoints públicos de las colecciones, un elemento por variante.
### Análisis Avanzado y Navegación
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # y así sucesivamente...
# Encuentra elementos por contenido de texto
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Navegación avanzada
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Selectores encadenados
+quote_texts = page.css('.quote').css('.text::text').getall() # Selectores encadenados
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Relaciones y similitud de elementos
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
Puedes usar el parser directamente si no necesitas obtener sitios web, como se muestra a continuación:
```python
@@ -424,22 +433,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` es consciente del contexto y puede funcionar tanto en patrones sync/async
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# Uso de sesión async
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` se adapta al contexto y funciona tanto con sync como con async
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # Opcional - El estado del pool de pestañas del navegador (ocupado/libre/error)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI y Shell Interactivo
@@ -450,13 +455,14 @@ Scrapling incluye una poderosa interfaz de línea de comandos:
Lanzar el Shell interactivo de Web Scraping
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Extraer páginas a un archivo directamente sin programar (Extrae el contenido dentro de la etiqueta `body` por defecto). Si el archivo de salida termina con `.txt`, entonces se extraerá el contenido de texto del objetivo. Si termina con `.md`, será una representación Markdown del contenido HTML; si termina con `.html`, será el contenido HTML en sí mismo.
+Extrae páginas directamente a un archivo sin programar. Usa `--css-selector` para elegir el contenido o `--ai-targeted` para seleccionar y limpiar el contenido de `body`. Usa `.txt` para texto, `.md` para Markdown o `.html` para HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos los elementos que coinciden con el selector CSS '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -477,7 +483,7 @@ Scrapling no solo es potente, también es ultrarrápido. Los siguientes benchmar
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -487,8 +493,8 @@ Las capacidades de búsqueda adaptativa de elementos de Scrapling superan signif
| Biblioteca | Tiempo (ms) | vs Scrapling |
|-------------|:-----------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> Todos los benchmarks representan promedios de más de 100 ejecuciones. Ver [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) para la metodología.
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Recuerda que necesitas instalar las dependencias del navegador con `scrapling install` después de cualquiera de estos extras (si no lo hiciste ya)
+ Ejecuta `scrapling install` cuando necesites fetchers de navegador. Las solicitudes HTTP no requieren instalar un navegador.
### Docker
También puedes instalar una imagen Docker con todos los extras y navegadores con el siguiente comando desde DockerHub:
@@ -571,7 +577,7 @@ Si has utilizado nuestra biblioteca con fines de investigación, por favor cíta
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_FR.md b/docs/README_FR.md
index b533021..b6f5944 100644
--- a/docs/README_FR.md
+++ b/docs/README_FR.md
@@ -57,10 +57,10 @@ Utilisez-le depuis Python, le terminal ou votre agent IA grâce à son serveur M
## Ce que vous pouvez faire avec Scrapling
-- **Récupérer des pages et interagir** : Requêtes HTTP rapides, navigateurs furtifs, rendu JavaScript, gestion de Cloudflare Turnstile/Interstitial, sessions persistantes et capture des appels API en arrière-plan.
-- **Analyser et extraire** : CSS, XPath, recherche par texte et expressions régulières, suivi adaptatif des éléments, recherche d'éléments similaires et sortie en Markdown propre.
-- **Crawler et exporter** : Spiders concurrents, rotation de proxys, réessais, AutoThrottle, pause/reprise, streaming, modèles prêts à l'emploi et exports JSON/JSONL/CSV/XML.
-- **Scraper avec l'IA ou depuis votre terminal** : Un serveur MCP pour que les agents IA naviguent et extraient des données, un Agent Skill, des commandes CLI et un shell interactif.
+- **[Récupérer des pages et interagir](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)** : Requêtes HTTP rapides, navigateurs furtifs, rendu JavaScript, gestion de Cloudflare Turnstile/Interstitial, sessions persistantes et capture des appels API en arrière-plan.
+- **[Analyser et extraire](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)** : CSS, XPath, recherche par texte et expressions régulières, suivi adaptatif des éléments, recherche d'éléments similaires et sortie en Markdown propre.
+- **[Crawler et exporter](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)** : Spiders concurrents, rotation de proxys, réessais, AutoThrottle, pause/reprise, streaming, modèles prêts à l'emploi et exports JSON/JSONL/CSV/XML.
+- **[Scraper avec l'IA ou depuis votre terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)** : Un serveur MCP pour que les agents IA naviguent et extraient des données, un Agent Skill, des commandes CLI et un shell interactif.
Commencez par une seule page :
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Ou montez en charge vers des crawls complets :
+Ou montez en charge avec un Spider :
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+Cela affiche `10` et enregistre les citations et leurs auteurs dans `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **API Spider à la Scrapy** : Définissez des spiders avec `start_urls`, des callbacks async `parse` et des objets `Request`/`Response`.
- ⚡ **Crawling concurrent** : Limites de concurrence configurables, throttling par domaine et délais de téléchargement.
- 🔄 **Support multi-sessions** : Interface unifiée pour les requêtes HTTP et les navigateurs headless furtifs dans un seul spider - routez les requêtes vers différentes sessions par ID.
-- 💾 **Pause & Reprise** : Persistance du crawl basée sur des checkpoints. Appuyez sur Ctrl+C pour un arrêt gracieux ; redémarrez pour reprendre là où vous vous étiez arrêté.
+- 💾 **Pause et reprise** : Enregistrez les requêtes en attente et les URL déjà vues avec `crawldir`. Arrêtez proprement avec Ctrl+C, puis relancez avec le même dossier.
- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel - idéal pour les UI, pipelines et crawls de longue durée.
- 🛡️ **Détection des requêtes bloquées** : Détection automatique et réessai des requêtes bloquées avec une logique personnalisable.
- 🚦 **AutoThrottle** : Fini les délais devinés. Le spider ajuste tout seul le délai de chaque domaine selon la vitesse de réponse du site, le double (ou attend ce que demande `Retry-After`) dès que le site se met à vous bloquer ou à vous limiter, et réaccélère quand cela s'arrête.
- 🤖 **Conformité robots.txt** : Flag optionnel `robots_txt_obey` qui respecte les directives `Disallow`, `Crawl-delay` et `Request-rate` avec mise en cache par domaine.
- 🧪 **Mode développement** : Mettez les réponses en cache sur le disque lors de la première exécution et rejouez-les lors des exécutions suivantes - itérez sur votre logique `parse()` sans solliciter à nouveau les serveurs cibles.
-- 🧩 **Modèles de Spider prêts à l'emploi** : Évitez le code répétitif avec `CrawlSpider` pour le suivi de liens par règles, `SitemapSpider` pour les crawls pilotés par sitemap/robots.txt, et `ShopifySpider` pour récupérer tous les produits de n'importe quelle boutique Shopify via son API JSON, un item par variante.
+- 🧩 **Modèles de Spider prêts à l'emploi** : Évitez le code répétitif avec `CrawlSpider` pour le suivi de liens par règles, `SitemapSpider` pour les crawls pilotés par sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` pour parcourir des flux XML/RSS et CSV et `ShopifySpider` pour collecter les produits des boutiques Shopify publiques, un élément par variante.
- 🔗 **Extraction de liens** : Un primitif `LinkExtractor` autonome avec motifs allow/deny, filtres de domaines, délimitation CSS/XPath, filtrage des extensions et canonicalisation - utilisable dans les modèles ou seul.
- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou les exports JSON/JSONL/CSV/XML intégrés avec `result.items.to_json()`, `to_jsonl()`, `to_csv()` et `to_xml()`.
### Récupération avancée de sites web avec support de sessions
- **Requêtes HTTP** : Requêtes HTTP rapides et furtives avec la classe `Fetcher`. Peut imiter l'empreinte TLS des navigateurs, les headers et utiliser HTTP/3.
- **Chargement dynamique** : Récupérez des sites web dynamiques avec une automatisation complète du navigateur via la classe `StealthyFetcher` supportant Chromium de Playwright et Google Chrome.
-- **Contournement anti-bot** : Capacités de furtivité avancées avec `StealthyFetcher` et usurpation d'empreinte. Peut facilement contourner tous les types de Turnstile/Interstitial de Cloudflare avec l'automatisation.
+- **Contournement anti-bot** : Capacités de furtivité avancées avec `StealthyFetcher` et falsification des empreintes. Contourne facilement tous les types de Turnstile/Interstitial de Cloudflare grâce à l'automatisation.
- **Gestion de sessions** : Support de sessions persistantes avec les classes `FetcherSession` et `StealthySession` pour la gestion des cookies et de l'état entre les requêtes.
- **Rotation de proxy** : `ProxyRotator` intégré avec des stratégies de rotation cycliques ou personnalisées sur tous les types de sessions, plus des surcharges de proxy par requête.
- **Blocage de domaines et publicités** : Bloquez les requêtes vers des domaines spécifiques (et leurs sous-domaines) ou activez le blocage de publicités intégré (~3 500 domaines publicitaires/traceurs connus) dans les fetchers basés sur navigateur.
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **Trouver des éléments similaires** : Localisez automatiquement des éléments similaires aux éléments trouvés.
### Fonctionnalités IA
-- 🤖 **Serveur MCP** : Laissez les chatbots et agents IA (Claude/Cursor/etc.) scraper via Scrapling avec des outils ponctuels ou basés sur des sessions couvrant les requêtes HTTP simples (toute méthode) et les chargements par navigateur furtif qui contournent Cloudflare. Les pages sont réduites avec des sélecteurs CSS et nettoyées du contenu d'injection de prompt avant que l'IA ne les voie, donc l'agent lit moins, coûte moins et ne peut pas être détourné par du texte caché. Captures d'écran, navigateurs distants via CDP et transport HTTP sécurisé par défaut sont inclus. ([vidéo de démonstration](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Serveur MCP** : Laissez les chatbots et agents IA (Claude/Cursor/etc.) scraper via Scrapling avec les requêtes HTTP, les actions du navigateur, les captures d’écran, les connexions enregistrées et les réponses API en arrière-plan. Il utilise notre navigateur furtif avec les bons outils pour permettre aux modèles IA adaptés de contourner tous les CAPTCHA rencontrés pendant leur travail (sans parler de notre solveur Cloudflare). Il combine de nombreuses méthodes pour réduire le contenu envoyé au modèle, bloquer les publicités et le protéger contre les injections de prompts. Les navigateurs distants via CDP et le transport HTTP avec options d’authentification sont inclus. ([Vidéo de démonstration](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill** : Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) prêt à installer qui enseigne toute la bibliothèque aux agents de codage, pour que le code qu'ils écrivent avec Scrapling corresponde à l'API actuelle au lieu d'être deviné.
- 📚 **Markdown prêt pour le RAG** : Transformez n'importe quelle page en Markdown propre, assaini et prêt pour les LLM en une ligne (`page.markdown()`), ou crawlez un site entier en un corpus Markdown avec le template `ContentSpider`, le tout sans LLM dans la boucle. ([documentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Garder
data = page.css('#padded_content a').getall()
# Ou utiliser le style requête ponctuelle : ouvre le navigateur pour cette requête, puis le ferme après
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Automatisation complète du navigateur
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
Construisez des crawlers complets avec des requêtes concurrentes, plusieurs types de sessions et pause/reprise :
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,7 +377,7 @@ Mettez en pause et reprenez les longs crawls avec des checkpoints en lançant le
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Appuyez sur Ctrl+C pour mettre en pause gracieusement - la progression est sauvegardée automatiquement. Plus tard, lorsque vous relancez le spider, passez le même `crawldir`, et il reprendra là où il s'était arrêté.
+Avec `crawldir`, Ctrl+C enregistre les requêtes en attente et les URL déjà vues. Relancez avec le même `crawldir` pour reprendre. Enregistrez les données extraites séparément.
Ou évitez complètement d'écrire la logique de crawl grâce aux modèles prêts à l'emploi, par exemple pour récupérer tout le catalogue d'une boutique Shopify :
```python
@@ -379,8 +386,9 @@ from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # Tous les produits de la boutique, un item par variante
+result = MyStore().start() # Tous les produits de la boutique, un élément par variante
```
+Le modèle lit les produits exposés par les endpoints publics des collections, un élément par variante.
### Parsing avancé & Navigation
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # et ainsi de suite...
# Trouver un élément par contenu textuel
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Navigation avancée
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Sélecteurs chaînés
+quote_texts = page.css('.quote').css('.text::text').getall() # Sélecteurs chaînés
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Relations et similarité entre éléments
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
Vous pouvez utiliser le parser directement si vous ne souhaitez pas récupérer de sites web, comme ci-dessous :
```python
@@ -424,22 +433,18 @@ Et cela fonctionne exactement de la même manière !
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` est sensible au contexte et peut fonctionner en mode sync comme async
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# Utilisation de session async
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` s'adapte au contexte et fonctionne en mode sync comme async
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # Optionnel - Le statut du pool d'onglets du navigateur (occupé/libre/erreur)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI & Shell interactif
@@ -450,13 +455,14 @@ Scrapling inclut une interface en ligne de commande puissante :
Lancer le shell interactif de Web Scraping
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Extraire des pages directement dans un fichier sans programmation (extrait par défaut le contenu de la balise `body`). Si le fichier de sortie se termine par `.txt`, le contenu textuel de la cible sera extrait. S'il se termine par `.md`, ce sera une représentation Markdown du contenu HTML ; s'il se termine par `.html`, ce sera le contenu HTML lui-même.
+Extrayez des pages directement dans un fichier sans programmation. Utilisez `--css-selector` pour choisir le contenu ou `--ai-targeted` pour sélectionner et nettoyer le contenu de `body`. Utilisez `.txt` pour du texte, `.md` pour du Markdown ou `.html` pour du HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Tous les éléments correspondant au sélecteur CSS '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- N'oubliez pas que vous devez installer les dépendances navigateur avec `scrapling install` après l'un de ces extras (si vous ne l'avez pas déjà fait)
+ Lancez `scrapling install` si vous utilisez les fetchers avec navigateur. Les requêtes HTTP ne nécessitent pas de navigateur.
### Docker
Vous pouvez également installer une image Docker avec tous les extras et navigateurs avec la commande suivante depuis DockerHub :
@@ -571,7 +577,7 @@ Si vous avez utilisé notre bibliothèque à des fins de recherche, veuillez nou
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_JP.md b/docs/README_JP.md
index d687186..7b58b26 100644
--- a/docs/README_JP.md
+++ b/docs/README_JP.md
@@ -57,10 +57,10 @@ Python やターミナルから、または MCP サーバーと Agent Skill を
## Scrapling でできること
-- **取得と操作**:高速な HTTP リクエスト、ステルスブラウザ、JavaScript のレンダリング、Cloudflare Turnstile/Interstitial への対応、セッションの維持、バックグラウンドの API 通信の記録。
-- **パースと抽出**:CSS、XPath、テキストと正規表現による検索、適応型の要素追跡、類似要素のマッチング、読みやすい Markdown 出力。
-- **クロールとエクスポート**:並行クロール、プロキシローテーション、リトライ、AutoThrottle、一時停止/再開、ストリーミング、すぐに使える Spider テンプレート、JSON/JSONL/CSV/XML エクスポート。
-- **AI やターミナルからスクレイピング**:AI エージェントがウェブを閲覧してデータを抽出するための MCP サーバー、Agent Skill、CLI コマンド、インタラクティブ Shell。
+- **[取得と操作](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**:高速な HTTP リクエスト、ステルスブラウザ、JavaScript のレンダリング、Cloudflare Turnstile/Interstitial への対応、セッションの維持、バックグラウンドの API 通信の記録。
+- **[パースと抽出](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**:CSS、XPath、テキストと正規表現による検索、適応型の要素追跡、類似要素のマッチング、読みやすい Markdown 出力。
+- **[クロールとエクスポート](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**:並行クロール、プロキシローテーション、リトライ、AutoThrottle、一時停止/再開、ストリーミング、すぐに使える Spider テンプレート、JSON/JSONL/CSV/XML エクスポート。
+- **[AI やターミナルからスクレイピング](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**:AI エージェントがウェブを閲覧してデータを抽出するための MCP サーバー、Agent Skill、CLI コマンド、インタラクティブ Shell。
まずは 1 ページから:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-または本格的なクロールへスケールアップ:
+または Spider でスケールアップ:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+`10` が表示され、引用と著者が `quotes.json` に保存されます。
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **Scrapy 風の Spider API**:`start_urls`、async `parse` callback、`Request`/`Response` オブジェクトで Spider を定義。
- ⚡ **並行クロール**:設定可能な並行数制限、ドメインごとのスロットリング、ダウンロード遅延。
- 🔄 **マルチ Session サポート**:HTTP リクエストとステルスヘッドレスブラウザの統一インターフェース - ID によって異なる Session にリクエストをルーティング。
-- 💾 **Pause & Resume**:Checkpoint ベースのクロール永続化。Ctrl+C で正常にシャットダウン;再起動すると中断したところから再開。
+- 💾 **一時停止と再開**:`crawldir` で保留中のリクエストと訪問済み URL を保存します。Ctrl+C で正常終了し、同じディレクトリで再起動します。
- 📡 **Streaming モード**:`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 - UI、パイプライン、長時間実行クロールに最適。
- 🛡️ **ブロックされたリクエストの検出**:カスタマイズ可能なロジックによるブロックされたリクエストの自動検出とリトライ。
- 🚦 **AutoThrottle**:待ち時間を勘で決める必要はありません。Spider がサイトの応答速度からドメインごとの待ち時間を自動調整し、ブロックやレート制限が始まると待ち時間を 2 倍に(または `Retry-After` が求める時間だけ待機)して、収まればまた加速します。
- 🤖 **robots.txt 準拠**:オプションの `robots_txt_obey` フラグで `Disallow`、`Crawl-delay`、`Request-rate` ディレクティブをドメインごとのキャッシュで遵守。
- 🧪 **開発モード**:初回実行時にレスポンスをディスクにキャッシュし、以降の実行ではそれを再生 - ターゲットサーバーに再リクエストすることなく `parse()` ロジックを反復開発できます。
-- 🧩 **すぐに使える Spider テンプレート**:定型コードは不要です。ルールベースのリンク追跡には `CrawlSpider`、sitemap/robots.txt を起点としたクロールには `SitemapSpider`、XML/RSS や CSV フィードの走査には `XMLFeedSpider`/`CSVFeedSpider`、そして `ShopifySpider` を使えば任意の Shopify ストアの全商品を JSON API 経由で取得できます(バリエーションごとに 1 アイテム)。
+- 🧩 **すぐに使える Spider テンプレート**:定型コードは不要です。ルールベースのリンク追跡には `CrawlSpider`、sitemap/robots.txt を起点としたクロールには `SitemapSpider`、XML/RSS や CSV フィードの走査には `XMLFeedSpider`/`CSVFeedSpider`、そして `ShopifySpider` で公開されている Shopify ストアの商品を取得できます(バリエーションごとに 1 件)。
- 🔗 **リンク抽出**:allow/deny パターン、ドメインフィルタ、CSS/XPath による範囲指定、拡張子フィルタ、正規化に対応した独立した `LinkExtractor` を提供。テンプレート内でも単体でも使用できます。
- 📦 **組み込みエクスポート**:フックや独自のパイプライン、または組み込みの JSON/JSONL/CSV/XML で結果をエクスポート。`result.items.to_json()`、`to_jsonl()`、`to_csv()`、`to_xml()` が使えます。
### Session サポート付き高度なウェブサイト取得
- **HTTP リクエスト**:`Fetcher` クラスで高速かつステルスな HTTP リクエスト。ブラウザの TLS fingerprint、ヘッダーを模倣し、HTTP/3 を使用可能。
- **動的読み込み**:Playwright の Chromium と Google Chrome をサポートする `StealthyFetcher` クラスによる完全なブラウザ自動化で動的ウェブサイトを取得。
-- **アンチボット回避**:`StealthyFetcher` と fingerprint 偽装による高度なステルス機能。自動化で Cloudflare の Turnstile/Interstitial のすべてのタイプを簡単に回避。
+- **アンチボット回避**:`StealthyFetcher` の高度なステルス機能とフィンガープリント偽装。あらゆる種類の Cloudflare Turnstile/Interstitial を自動で簡単に回避できます。
- **Session 管理**:リクエスト間で Cookie と状態を管理するための `FetcherSession`、`StealthySession` クラスによる永続的な Session サポート。
- **Proxy 回転**:すべての Session タイプに対応したラウンドロビンまたはカスタム戦略の組み込み `ProxyRotator`、さらにリクエストごとの Proxy オーバーライド。
- **ドメイン&広告ブロック**:ブラウザベースの Fetcher で特定のドメイン(およびそのサブドメイン)へのリクエストをブロック、または内蔵広告ブロック(約3,500の既知の広告/トラッカードメイン)を有効化。
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **類似要素の検出**:見つかった要素に類似した要素を自動的に特定。
### AI 機能
-- 🤖 **MCP サーバー**:AI チャットボットやエージェント(Claude/Cursor など)が Scrapling を通じてスクレイピングできるようにします。単発ツールとセッションベースのツールがあり、通常の HTTP リクエスト(任意のメソッド)と、Cloudflare を回避するステルスブラウザ取得をカバーします。ページは AI に渡す前に CSS セレクターで絞り込まれ、プロンプトインジェクションの内容が除去されるため、エージェントは読む量もコストも減り、隠しテキストに乗っ取られることもありません。スクリーンショット、CDP 経由のリモートブラウザ、デフォルトで安全な HTTP トランスポートも含まれます。([デモ動画](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP サーバー**:AI チャットボットやエージェント(Claude/Cursor など)が Scrapling を通じてスクレイピングできるようにします。HTTP リクエスト、ブラウザ操作、スクリーンショット、保存済みログイン状態、バックグラウンド API レスポンスを利用できます。独自のステルスブラウザと適切なツールで、十分な能力を持つ AI モデルが作業中に遭遇するどんな CAPTCHA でも回避できるようにします(もちろん、独自の Cloudflare ソルバーもあります)。モデルに送る内容の削減、広告のブロック、プロンプトインジェクションからの保護にさまざまな手法を使います。CDP 経由のリモートブラウザと認証オプション付き HTTP トランスポートにも対応します。([デモ動画](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:ライブラリ全体をコーディングエージェントに教える、インストールするだけの [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)。エージェントが Scrapling で書くコードが、推測ではなく現在の API に沿ったものになります。
- 📚 **RAG 対応の Markdown**:1 行(`page.markdown()`)で任意のページをクリーンでサニタイズ済みの LLM 向け Markdown に変換したり、`ContentSpider` テンプレートでサイト全体を Markdown コーパスとしてクロールしたりできます。処理に LLM は一切使いません。([ドキュメント](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # 完了
data = page.css('#padded_content a').getall()
# または一回限りのリクエストスタイル、このリクエストのためにブラウザを開き、完了後に閉じる
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
完全なブラウザ自動化
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spider
並行リクエスト、複数の Session タイプ、Pause & Resume を備えた本格的なクローラーを構築:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,17 +377,18 @@ Checkpoint を使用して長時間のクロールをPause & Resume:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Ctrl+C を押すと正常に一時停止し、進捗は自動的に保存されます。後で Spider を再度起動する際に同じ`crawldir`を渡すと、中断したところから再開します。
+`crawldir` を設定して Ctrl+C を押すと、保留中のリクエストと訪問済み URL が保存されます。同じ `crawldir` で再起動すると再開できます。抽出データは別途保存してください。
-あるいは、すぐに使えるテンプレートでクロールのロジックを書くこと自体を省略できます。たとえば Shopify ストアのカタログ全体を取得する場合:
+あるいは、すぐに使えるテンプレートでクロールのロジックを書くこと自体を省略できます。たとえば Shopify ストアのカタログを丸ごと取得する場合:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # ストア内の全商品、バリエーションごとに 1 アイテム
+result = MyStore().start() # ストア内の全商品を、バリエーションごとに 1 件取得
```
+テンプレートは公開コレクション API が提供する商品を、バリエーションごとに 1 件取得します。
### 高度なパースとナビゲーション
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # など...
# テキスト内容で要素を検索
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# 高度なナビゲーション
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # チェーンセレクタ
+quote_texts = page.css('.quote').css('.text::text').getall() # チェーンセレクタ
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# 要素の関連性と類似性
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
ウェブサイトを取得せずにパーサーをすぐに使用することもできます:
```python
@@ -424,22 +433,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` はコンテキストアウェアで、同期/非同期両方のパターンで動作可能
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# 非同期 Session の使用
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` はコンテキストに応じて同期・非同期の両方で動作します
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # オプション - ブラウザタブプールのステータス(ビジー/フリー/エラー)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI とインタラクティブ Shell
@@ -450,13 +455,14 @@ Scrapling には強力なコマンドラインインターフェースが含ま
インタラクティブ Web Scraping Shell を起動
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-プログラミングせずに直接ページをファイルに抽出(デフォルトで`body`タグ内のコンテンツを抽出)。出力ファイルが`.txt`で終わる場合、ターゲットのテキストコンテンツが抽出されます。`.md`で終わる場合、HTML コンテンツの Markdown 表現になります。`.html` で終わる場合、HTML コンテンツそのものになります。
+プログラミングなしでページをファイルに抽出できます。`--css-selector` で内容を選ぶか、`--ai-targeted` で `body` の内容を選択・クリーニングします。テキストは `.txt`、Markdown は `.md`、HTML は `.html` を使います。
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSS セレクタ'#fromSkipToProducts'に一致するすべての要素
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -477,7 +483,7 @@ Scrapling は強力であるだけでなく、超高速です。以下のベン
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -487,8 +493,8 @@ Scrapling の適応型要素検索機能は代替手段を大幅に上回りま
| ライブラリ | 時間 (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> すべてのベンチマークは 100 回以上の実行の平均を表します。方法論については[benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py)を参照してください。
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- これらの追加機能のいずれかの後(まだインストールしていない場合)、`scrapling install`でブラウザの依存関係をインストールする必要があることを忘れないでください
+ ブラウザ fetcher を使う場合は `scrapling install` を実行してください。HTTP リクエストにはブラウザのインストールは不要です。
### Docker
DockerHub から次のコマンドですべての追加機能とブラウザを含む Docker イメージをインストールすることもできます:
@@ -571,7 +577,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_KR.md b/docs/README_KR.md
index 9020b21..212b5e4 100644
--- a/docs/README_KR.md
+++ b/docs/README_KR.md
@@ -57,10 +57,10 @@ Python이나 터미널에서, 또는 MCP 서버와 Agent Skill을 통해 AI 에
## Scrapling으로 할 수 있는 일
-- **가져오기와 조작**: 빠른 HTTP 요청, 스텔스 브라우저, JavaScript 렌더링, Cloudflare Turnstile/Interstitial 처리, 세션 유지, 백그라운드 API 통신 캡처.
-- **파싱과 추출**: CSS, XPath, 텍스트와 정규식 검색, 적응형 요소 추적, 유사 요소 매칭, 깔끔한 Markdown 출력.
-- **크롤링과 내보내기**: 동시 크롤링, 프록시 로테이션, 재시도, AutoThrottle, 일시정지/재개, 스트리밍, 바로 쓸 수 있는 Spider 템플릿, JSON/JSONL/CSV/XML 내보내기.
-- **AI나 터미널에서 스크래핑**: AI 에이전트가 웹을 탐색하고 데이터를 추출할 수 있는 MCP 서버, Agent Skill, CLI 명령어, 인터랙티브 Shell.
+- **[가져오기와 조작](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: 빠른 HTTP 요청, 스텔스 브라우저, JavaScript 렌더링, Cloudflare Turnstile/Interstitial 처리, 세션 유지, 백그라운드 API 통신 캡처.
+- **[파싱과 추출](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, 텍스트와 정규식 검색, 적응형 요소 추적, 유사 요소 매칭, 깔끔한 Markdown 출력.
+- **[크롤링과 내보내기](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: 동시 크롤링, 프록시 로테이션, 재시도, AutoThrottle, 일시정지/재개, 스트리밍, 바로 쓸 수 있는 Spider 템플릿, JSON/JSONL/CSV/XML 내보내기.
+- **[AI나 터미널에서 스크래핑](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: AI 에이전트가 웹을 탐색하고 데이터를 추출할 수 있는 MCP 서버, Agent Skill, CLI 명령어, 인터랙티브 Shell.
먼저 한 페이지부터 시작하세요:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-또는 본격적인 크롤링으로 확장:
+또는 Spider로 크롤링을 확장하세요:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+`10`이 출력되고 명언과 저자가 `quotes.json`에 저장됩니다.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -229,20 +236,20 @@ MySpider().start()
- 🕷️ **Scrapy 스타일 Spider API**: `start_urls`, 비동기 `parse` 콜백, `Request`/`Response` 객체로 Spider를 정의합니다.
- ⚡ **동시 크롤링**: 설정 가능한 동시 요청 수 제한, 도메인별 스로틀링, 다운로드 딜레이를 지원합니다.
- 🔄 **멀티 세션 지원**: HTTP 요청과 스텔스 헤드리스 브라우저를 하나의 인터페이스로 통합 - ID로 요청을 다른 세션에 라우팅합니다.
-- 💾 **일시정지 & 재개**: 체크포인트 기반의 크롤링 영속화. Ctrl+C로 정상 종료하고, 재시작하면 중단된 지점부터 이어갑니다.
+- 💾 **일시정지와 재개**: `crawldir`로 대기 중인 요청과 방문한 URL을 저장합니다. Ctrl+C로 정상 종료한 뒤 같은 디렉터리로 다시 시작하세요.
- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 - UI, 파이프라인, 장시간 크롤링에 적합합니다.
- 🛡️ **차단된 요청 감지**: 커스텀 로직을 통한 차단된 요청의 자동 감지 및 재시도를 지원합니다.
- 🚦 **AutoThrottle**: 지연 시간을 추측할 필요가 없습니다. Spider가 사이트의 응답 속도에 맞춰 도메인별 지연 시간을 스스로 조정하고, 차단이나 속도 제한이 시작되면 지연을 두 배로 늘리거나 `Retry-After`가 요구하는 만큼 기다렸다가, 상황이 풀리면 다시 속도를 올립니다.
- 🤖 **robots.txt 준수**: 선택적 `robots_txt_obey` 플래그로 `Disallow`, `Crawl-delay`, `Request-rate` 지시문을 도메인별 캐싱과 함께 준수합니다.
- 🧪 **개발 모드**: 첫 실행 시 응답을 디스크에 캐싱하고 이후 실행에서는 캐시된 응답을 재생합니다 - 대상 서버에 다시 요청하지 않고 `parse()` 로직을 반복 개발할 수 있습니다.
-- 🧩 **바로 쓰는 Spider 템플릿**: 반복적인 코드는 건너뛰세요. 규칙 기반 링크 추적에는 `CrawlSpider`, sitemap/robots.txt 기반 크롤링에는 `SitemapSpider`, XML/RSS 및 CSV 피드 순회에는 `XMLFeedSpider`/`CSVFeedSpider`, 그리고 `ShopifySpider`로 어떤 Shopify 스토어든 JSON API를 통해 모든 상품을 가져올 수 있습니다(변형별로 항목 하나).
+- 🧩 **바로 쓰는 Spider 템플릿**: 반복적인 코드는 건너뛰세요. 규칙 기반 링크 추적에는 `CrawlSpider`, sitemap/robots.txt 기반 크롤링에는 `SitemapSpider`, XML/RSS 및 CSV 피드 순회에는 `XMLFeedSpider`/`CSVFeedSpider`, 그리고 `ShopifySpider`로 공개된 Shopify 스토어의 상품을 변형당 한 항목씩 수집할 수 있습니다.
- 🔗 **링크 추출**: allow/deny 패턴, 도메인 필터, CSS/XPath 범위 지정, 확장자 필터링, 정규화를 지원하는 독립형 `LinkExtractor`. 템플릿 안에서도, 단독으로도 사용할 수 있습니다.
- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL/CSV/XML로 결과를 내보냅니다. `result.items.to_json()`, `to_jsonl()`, `to_csv()`, `to_xml()`을 사용합니다.
### 세션을 지원하는 고급 웹사이트 가져오기
- **HTTP 요청**: `Fetcher` 클래스로 빠르고 은밀한 HTTP 요청. 브라우저의 TLS fingerprint, 헤더를 모방하고, HTTP/3를 사용할 수 있습니다.
- **동적 로딩**: Playwright의 Chromium과 Google Chrome을 지원하는 `StealthyFetcher` 클래스로 완전한 브라우저 자동화를 통해 동적 웹사이트를 가져옵니다.
-- **안티봇 우회**: `StealthyFetcher`와 fingerprint 위장을 통한 고급 스텔스 기능. 자동화로 모든 유형의 Cloudflare Turnstile/Interstitial을 손쉽게 우회합니다.
+- **안티봇 우회**: `StealthyFetcher`의 고급 스텔스 기능과 브라우저 지문 위장. 모든 유형의 Cloudflare Turnstile/Interstitial을 자동으로 쉽게 우회할 수 있습니다.
- **세션 관리**: `FetcherSession`, `StealthySession` 클래스로 요청 간 쿠키와 상태를 관리하는 영속적 세션을 지원합니다.
- **프록시 로테이션**: 모든 세션 타입에 대응하는 순환 또는 커스텀 전략의 내장 `ProxyRotator`와 요청별 프록시 오버라이드를 제공합니다.
- **도메인 및 광고 차단**: 브라우저 기반 Fetcher에서 특정 도메인(및 하위 도메인)으로의 요청을 차단하거나 내장 광고 차단(약 3,500개의 알려진 광고/트래커 도메인)을 활성화합니다.
@@ -257,7 +264,7 @@ MySpider().start()
- 🔍 **유사 요소 찾기**: 발견된 요소와 유사한 요소를 자동으로 찾아냅니다.
### AI 기능
-- 🤖 **MCP 서버**: AI 챗봇과 에이전트(Claude/Cursor 등)가 Scrapling을 통해 스크래핑하도록 해줍니다. 일회성 도구와 세션 기반 도구가 일반 HTTP 요청(모든 메서드)과 Cloudflare를 우회하는 스텔스 브라우저 가져오기를 모두 지원합니다. 페이지는 AI에 전달되기 전에 CSS 선택자로 범위가 좁혀지고 프롬프트 인젝션 콘텐츠가 제거되므로, 에이전트는 더 적게 읽고 더 적게 비용을 쓰며 숨겨진 텍스트에 탈취당하지 않습니다. 스크린샷, CDP를 통한 원격 브라우저, 기본적으로 안전한 HTTP 전송도 포함됩니다. ([데모 영상](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP 서버**: AI 챗봇과 에이전트(Claude/Cursor 등)가 Scrapling을 통해 스크래핑할 수 있습니다. HTTP 요청, 브라우저 작업, 스크린샷, 저장된 로그인 상태, 백그라운드 API 응답을 사용할 수 있습니다. 자체 스텔스 브라우저와 적절한 도구로, 충분한 능력을 갖춘 AI 모델이 작업 중 마주치는 어떤 CAPTCHA든 우회할 수 있게 합니다(자체 Cloudflare 솔버는 물론이고요). 여러 방법으로 모델에 보내는 내용을 줄이고, 광고를 차단하고, 프롬프트 인젝션으로부터 모델을 보호합니다. CDP 원격 브라우저와 인증 옵션이 있는 HTTP 전송도 지원합니다. ([데모 영상](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: 설치만 하면 되는 [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)이 코딩 에이전트에게 라이브러리 전체를 가르쳐, 에이전트가 Scrapling으로 작성하는 코드가 추측이 아니라 현재 API에 맞도록 해줍니다.
- 📚 **RAG용 Markdown**: 한 줄(`page.markdown()`)로 어떤 페이지든 깨끗하고 정제된 LLM용 Markdown으로 변환하거나, `ContentSpider` 템플릿으로 웹사이트 전체를 Markdown 코퍼스로 크롤링할 수 있습니다. 과정에 LLM을 전혀 사용하지 않습니다. ([문서](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -304,7 +311,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # 작업
data = page.css('#padded_content a').getall()
# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
완전한 브라우저 자동화
@@ -323,7 +330,7 @@ data = page.css('.quote .text::text').getall()
### Spider
동시 요청, 여러 세션 타입, 일시정지 & 재개를 갖춘 본격적인 크롤러 구축:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -370,17 +377,18 @@ class MultiSessionSpider(Spider):
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Ctrl+C를 누르면 정상적으로 일시정지되고, 진행 상황이 자동 저장됩니다. 이후 Spider를 다시 시작할 때 동일한 `crawldir`을 전달하면 중단된 지점부터 재개합니다.
+`crawldir`를 설정한 뒤 Ctrl+C를 누르면 대기 중인 요청과 방문한 URL이 저장됩니다. 같은 `crawldir`로 다시 시작하면 이어서 실행됩니다. 추출한 데이터는 따로 저장하세요.
-또는 바로 쓰는 템플릿으로 크롤링 로직 작성 자체를 건너뛸 수 있습니다. 예를 들어 Shopify 스토어의 전체 카탈로그를 가져오려면:
+또는 바로 쓰는 템플릿으로 크롤링 로직 작성 자체를 건너뛸 수 있습니다. 예를 들어 Shopify 스토어의 상품 카탈로그 전체를 가져오려면:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # 스토어의 모든 상품, 변형별로 항목 하나
+result = MyStore().start() # 스토어의 모든 상품을 변형당 한 항목씩 수집
```
+템플릿은 공개 컬렉션 API에 노출된 상품을 변형당 한 항목씩 읽습니다.
### 고급 파싱 & 내비게이션
```python
@@ -398,18 +406,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # 등등...
# 텍스트 내용으로 요소 찾기
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# 고급 내비게이션
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # 체이닝 셀렉터
+quote_texts = page.css('.quote').css('.text::text').getall() # 체이닝 셀렉터
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# 요소 관계와 유사도
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
웹사이트를 가져오지 않고 파서를 바로 사용할 수도 있습니다:
```python
@@ -424,22 +433,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession`은 컨텍스트 인식이 가능하며 동기/비동기 패턴 모두에서 작동
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# 비동기 세션 사용
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession`은 컨텍스트에 따라 동기/비동기 방식 모두로 동작합니다
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # 선택 사항 - 브라우저 탭 풀 상태 (사용 중/유휴/에러)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI & 인터랙티브 Shell
@@ -450,13 +455,14 @@ Scrapling에는 강력한 커맨드라인 인터페이스가 포함되어 있습
인터랙티브 Web Scraping Shell 실행
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-프로그래밍 없이 페이지를 파일로 바로 추출합니다 (기본적으로 `body` 태그 내부의 콘텐츠를 추출). 출력 파일이 `.txt`로 끝나면 대상의 텍스트 콘텐츠가 추출됩니다. `.md`로 끝나면 HTML 콘텐츠의 Markdown 표현이 됩니다. `.html`로 끝나면 HTML 콘텐츠 자체가 됩니다.
+프로그래밍 없이 페이지를 파일로 바로 추출합니다. `--css-selector`로 내용을 고르거나 `--ai-targeted`로 `body` 내용을 선택하고 정리하세요. 텍스트는 `.txt`, Markdown은 `.md`, HTML은 `.html`을 사용하세요.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSS selector '#fromSkipToProducts'에 매칭되는 모든 요소
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -477,7 +483,7 @@ Scrapling은 강력할 뿐만 아니라 초고속입니다. 아래 벤치마크
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -487,8 +493,8 @@ Scrapling의 적응형 요소 찾기 기능은 대안들을 크게 앞섭니다:
| Library | Time (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> 모든 벤치마크는 100회 이상 실행의 평균입니다. 측정 방법은 [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py)를 참조하세요.
@@ -541,7 +547,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- 위 추가 기능을 설치한 후에도 (아직 하지 않았다면) `scrapling install`로 브라우저 의존성을 설치해야 합니다.
+ 브라우저 fetcher가 필요하면 `scrapling install`을 실행하세요. HTTP 요청은 브라우저 설치가 필요하지 않습니다.
### Docker
DockerHub에서 모든 추가 기능과 브라우저가 포함된 Docker 이미지를 설치할 수도 있습니다:
@@ -571,7 +577,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.md
index 05bf328..2d4a584 100644
--- a/docs/README_PT_BR.md
+++ b/docs/README_PT_BR.md
@@ -59,10 +59,10 @@ Use pelo Python, pelo terminal ou pelo seu agente de IA com o servidor MCP e o A
## O que você pode fazer com Scrapling
-- **Buscar páginas e interagir**: Requisições HTTP rápidas, navegadores furtivos, renderização de JavaScript, tratamento de Cloudflare Turnstile/Interstitial, sessões persistentes e captura de chamadas de API em segundo plano.
-- **Analisar e extrair**: CSS, XPath, busca por texto e expressões regulares, rastreamento adaptativo de elementos, busca de elementos semelhantes e saída em Markdown limpo.
-- **Rastrear e exportar**: Spiders concorrentes, rotação de proxies, retries, AutoThrottle, pausa/retomada, streaming, templates prontos e exportação para JSON/JSONL/CSV/XML.
-- **Fazer scraping com IA ou pelo terminal**: Um servidor MCP para agentes de IA navegarem e extraírem dados, um Agent Skill, comandos CLI e um shell interativo.
+- **[Buscar páginas e interagir](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Requisições HTTP rápidas, navegadores furtivos, renderização de JavaScript, tratamento de Cloudflare Turnstile/Interstitial, sessões persistentes e captura de chamadas de API em segundo plano.
+- **[Analisar e extrair](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, busca por texto e expressões regulares, rastreamento adaptativo de elementos, busca de elementos semelhantes e saída em Markdown limpo.
+- **[Rastrear e exportar](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Spiders concorrentes, rotação de proxies, retries, AutoThrottle, pausa/retomada, streaming, templates prontos e exportação para JSON/JSONL/CSV/XML.
+- **[Fazer scraping com IA ou pelo terminal](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: Um servidor MCP para agentes de IA navegarem e extraírem dados, um Agent Skill, comandos CLI e um shell interativo.
Comece com uma única página:
@@ -74,22 +74,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Ou escale para crawls completos:
+Ou escale com um Spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+Isso imprime `10` e salva as citações e os autores em `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -231,20 +238,20 @@ MySpider().start()
- 🕷️ **API de Spider estilo Scrapy**: Defina spiders com `start_urls`, callbacks assíncronos `parse` e objetos `Request`/`Response`.
- ⚡ **Crawling Concorrente**: Limites de concorrência configuráveis, throttling por domínio e delays de download.
- 🔄 **Suporte Multi-Sessão**: Interface unificada para requisições HTTP e navegadores headless furtivos em uma única spider - direcione requisições para diferentes sessões por ID.
-- 💾 **Pausa e Retomada**: Persistência de crawl baseada em checkpoints. Pressione Ctrl+C para um encerramento gracioso; reinicie para continuar de onde parou.
+- 💾 **Pausa e retomada**: Salve requisições pendentes e URLs já vistas com `crawldir`. Pare com Ctrl+C e reinicie com o mesmo diretório.
- 📡 **Modo Streaming**: Faça streaming dos itens extraídos conforme chegam com `async for item in spider.stream()` e estatísticas em tempo real - ideal para UI, pipelines e crawls de longa duração.
- 🛡️ **Detecção de Requisições Bloqueadas**: Detecção automática e retry de requisições bloqueadas com lógica personalizável.
- 🚦 **AutoThrottle**: Pare de adivinhar os atrasos. A spider ajusta sozinha o atraso de cada domínio conforme a velocidade de resposta do site, dobra esse atraso (ou espera o que o `Retry-After` pedir) quando o site começa a bloquear ou limitar você, e volta a acelerar quando isso passa.
- 🤖 **Conformidade com robots.txt**: Flag opcional `robots_txt_obey` que respeita as diretivas `Disallow`, `Crawl-delay` e `Request-rate` com cache por domínio.
- 🧪 **Modo de Desenvolvimento**: Armazene respostas em disco na primeira execução e reproduza-as nas seguintes - itere sobre sua lógica de `parse()` sem reenviar requisições aos servidores-alvo.
-- 🧩 **Templates de Spider Prontos**: Pule o código repetitivo com `CrawlSpider` para seguir links por regras, `SitemapSpider` para crawls guiados por sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` para iterar feeds XML/RSS e CSV, e `ShopifySpider` para extrair todos os produtos de qualquer loja Shopify através da sua API JSON, um item por variação.
+- 🧩 **Templates de Spider Prontos**: Pule o código repetitivo com `CrawlSpider` para seguir links por regras, `SitemapSpider` para crawls guiados por sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` para iterar feeds XML/RSS e CSV, e `ShopifySpider` para coletar produtos de lojas públicas do Shopify, um item por variante.
- 🔗 **Extração de Links**: Um primitivo `LinkExtractor` independente com padrões allow/deny, filtros de domínio, delimitação por CSS/XPath, filtragem de extensões e canonicalização - use dentro dos templates ou sozinho.
- 📦 **Exportação Nativa**: Exporte resultados via hooks, seu próprio pipeline ou os exportadores nativos JSON/JSONL/CSV/XML com `result.items.to_json()`, `to_jsonl()`, `to_csv()` e `to_xml()`.
### Busca Avançada de Sites com Suporte a Sessões
- **Requisições HTTP**: Requisições HTTP rápidas e furtivas com a classe `Fetcher`. Pode imitar fingerprint TLS de navegadores, cabeçalhos e usar HTTP/3.
- **Carregamento Dinâmico**: Busque sites dinâmicos com automação completa de navegador através da classe `StealthyFetcher`, compatível com o Chromium do Playwright e o Google Chrome.
-- **Bypass Anti-Bot**: Capacidades avançadas de stealth com `StealthyFetcher` e spoofing de fingerprint. Pode contornar facilmente todos os tipos de Turnstile/Interstitial do Cloudflare com automação.
+- **Evasão anti-bot**: Recursos avançados de furtividade com `StealthyFetcher` e falsificação de fingerprints. Pode contornar facilmente todos os tipos de Turnstile/Interstitial da Cloudflare com automação.
- **Gerenciamento de Sessão**: Suporte a sessões persistentes com as classes `FetcherSession` e `StealthySession` para gerenciar cookies e estado entre requisições.
- **Rotação de Proxy**: `ProxyRotator` nativo com estratégias cíclicas ou personalizadas em todos os tipos de sessão, além de sobrescritas de proxy por requisição.
- **Bloqueio de Domínios e Anúncios**: Bloqueie requisições para domínios específicos (e seus subdomínios) ou habilite o bloqueio nativo de anúncios (~3.500 domínios conhecidos de anúncios/rastreadores) nos fetchers baseados em navegador.
@@ -259,7 +266,7 @@ MySpider().start()
- 🔍 **Encontrar Elementos Semelhantes**: Localize automaticamente elementos parecidos com os elementos encontrados.
### Recursos de IA
-- 🤖 **Servidor MCP**: Permita que chatbots e agentes de IA (Claude/Cursor/etc) façam scraping através do Scrapling com ferramentas de uso único ou baseadas em sessão, cobrindo requisições HTTP simples (qualquer método) e carregamentos com navegador furtivo que contornam o Cloudflare. As páginas são reduzidas com seletores CSS e limpas de conteúdo de injeção de prompt antes que a IA as veja, então o agente lê menos, custa menos e não pode ser sequestrado por texto oculto. Screenshots, navegadores remotos via CDP e um transporte HTTP seguro por padrão estão incluídos. ([vídeo demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **Servidor MCP**: Deixe chatbots e agentes de IA (Claude/Cursor/etc.) fazerem scraping pelo Scrapling com requisições HTTP, ações no navegador, capturas de tela, logins salvos e respostas de API em segundo plano. Usa nosso navegador furtivo com as ferramentas certas para que modelos de IA adequados contornem qualquer CAPTCHA que encontrem enquanto trabalham (sem falar no nosso resolvedor de Cloudflare). Usa vários métodos para reduzir o conteúdo enviado ao modelo, bloquear anúncios e protegê-lo contra Prompt-Injection. Inclui navegadores remotos por CDP e transporte HTTP com opções de autenticação. ([Vídeo de demonstração](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Um [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) pronto para instalar que ensina a biblioteca inteira aos agentes de codificação, para que o código que eles escrevem com Scrapling corresponda à API atual em vez de ser adivinhado.
- 📚 **Markdown pronto para RAG**: Transforme qualquer página em Markdown limpo, sanitizado e pronto para LLMs com uma linha (`page.markdown()`), ou rastreie um site inteiro em um corpus de Markdown com o template `ContentSpider`, tudo sem nenhum LLM no processo. ([documentação](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -306,7 +313,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Manten
data = page.css('#padded_content a').getall()
# Ou use o estilo de requisição avulsa, ele abre o navegador para esta requisição e o fecha ao finalizar
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Automação completa de navegador
@@ -325,7 +332,7 @@ data = page.css('.quote .text::text').getall()
### Spiders
Construa crawlers completos com requisições concorrentes, múltiplos tipos de sessão e pausa/retomada:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -372,17 +379,18 @@ Pause e retome crawls longos com checkpoints executando a spider assim:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Pressione Ctrl+C para pausar de forma graciosa - o progresso é salvo automaticamente. Depois, quando você iniciar a spider novamente, passe o mesmo `crawldir` e ela continuará de onde parou.
+Com `crawldir` definido, pressione Ctrl+C para salvar requisições pendentes e URLs já vistas. Reinicie com o mesmo `crawldir` para continuar. Salve os dados extraídos separadamente.
-Ou pule completamente a escrita da lógica de crawling com os templates prontos, como puxar o catálogo inteiro de uma loja Shopify:
+Ou pule completamente a escrita da lógica de crawling com os templates prontos, como extrair o catálogo inteiro de uma loja Shopify:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # Todos os produtos da loja, um item por variação
+result = MyStore().start() # Todos os produtos da loja, um item por variante
```
+O template lê os produtos expostos pelos endpoints públicos das coleções, um item por variante.
### Parsing Avançado e Navegação
```python
@@ -400,18 +408,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # e assim por diante...
# Encontre elementos por conteúdo de texto
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Navegação avançada
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Seletores encadeados
+quote_texts = page.css('.quote').css('.text::text').getall() # Seletores encadeados
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Relações e similaridade entre elementos
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
Você pode usar o parser imediatamente se não quiser buscar sites, como abaixo:
```python
@@ -426,22 +435,18 @@ E ele funciona exatamente da mesma maneira!
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` entende o contexto e funciona tanto em padrões sync quanto async
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` se adapta ao contexto e funciona tanto com sync quanto com async
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
-# Uso de sessão assíncrona
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
-
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
-
- print(session.get_pool_stats()) # Opcional - O estado do pool de abas do navegador (ocupada/livre/erro)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
+
+asyncio.run(main())
```
## CLI e Shell Interativo
@@ -452,13 +457,14 @@ O Scrapling inclui uma poderosa interface de linha de comando:
Inicie o shell interativo de Web Scraping
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Extraia páginas diretamente para um arquivo sem programar (por padrão, extrai o conteúdo dentro da tag `body`). Se o arquivo de saída terminar com `.txt`, então o conteúdo em texto do alvo será extraído. Se terminar com `.md`, será uma representação em Markdown do conteúdo HTML; se terminar com `.html`, será o próprio conteúdo HTML.
+Extraia páginas diretamente para um arquivo sem programar. Use `--css-selector` para escolher o conteúdo ou `--ai-targeted` para selecionar e limpar o conteúdo de `body`. Use `.txt` para texto, `.md` para Markdown ou `.html` para HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos os elementos que correspondem ao seletor CSS '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -479,7 +485,7 @@ O Scrapling não é apenas poderoso - ele também é extremamente rápido. Os be
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -489,8 +495,8 @@ Os recursos de localização adaptativa de elementos do Scrapling superam signif
| Biblioteca | Tempo (ms) | vs Scrapling |
|-------------|:----------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> Todos os benchmarks representam médias de 100+ execuções. Veja [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) para a metodologia.
@@ -543,7 +549,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Lembre-se de que você precisa instalar as dependências de navegador com `scrapling install` depois de qualquer um desses extras (caso ainda não tenha feito isso)
+ Execute `scrapling install` quando precisar dos fetchers de navegador. Requisições HTTP não precisam de um navegador instalado.
### Docker
Você também pode baixar uma imagem Docker com todos os extras e navegadores com o seguinte comando a partir do DockerHub:
@@ -573,7 +579,7 @@ Se você usou nossa biblioteca para fins de pesquisa, cite-nos com a seguinte re
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/README_RU.md b/docs/README_RU.md
index cac48df..9a1e9ce 100644
--- a/docs/README_RU.md
+++ b/docs/README_RU.md
@@ -57,10 +57,10 @@ Scrapling - это фреймворк для Web Scraping на Python, кото
## Что можно делать с Scrapling
-- **Загружать и взаимодействовать**: Быстрые HTTP-запросы, скрытные браузеры, рендеринг JavaScript, работа с Cloudflare Turnstile/Interstitial, постоянные сессии и запись фоновых API-запросов.
-- **Разбирать и извлекать**: CSS, XPath, поиск по тексту и регулярным выражениям, адаптивное отслеживание элементов, поиск похожих элементов и вывод в чистом Markdown.
-- **Обходить сайты и экспортировать данные**: Параллельные пауки, ротация прокси, повторные попытки, AutoThrottle, пауза и возобновление работы, Streaming, готовые шаблоны и экспорт в JSON/JSONL/CSV/XML.
-- **Собирать данные с ИИ или из терминала**: MCP-сервер для ИИ-агентов с возможностью просматривать сайты и извлекать данные, Agent Skill, команды CLI и интерактивная Shell.
+- **[Загружать и взаимодействовать](https://scrapling.readthedocs.io/en/latest/fetching/choosing.html)**: Быстрые HTTP-запросы, скрытные браузеры, рендеринг JavaScript, работа с Cloudflare Turnstile/Interstitial, постоянные сессии и запись фоновых API-запросов.
+- **[Разбирать и извлекать](https://scrapling.readthedocs.io/en/latest/parsing/selection.html)**: CSS, XPath, поиск по тексту и регулярным выражениям, адаптивное отслеживание элементов, поиск похожих элементов и вывод в чистом Markdown.
+- **[Обходить сайты и экспортировать данные](https://scrapling.readthedocs.io/en/latest/spiders/getting-started.html)**: Параллельные пауки, ротация прокси, повторные попытки, AutoThrottle, пауза и возобновление работы, Streaming, готовые шаблоны и экспорт в JSON/JSONL/CSV/XML.
+- **[Собирать данные с ИИ или из терминала](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)**: MCP-сервер для ИИ-агентов с возможностью просматривать сайты и извлекать данные, Agent Skill, команды CLI и интерактивная Shell.
Начните с одной страницы:
@@ -72,22 +72,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Или масштабируйте до полного обхода:
+Или масштабируйте сбор данных с помощью Spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+Код выводит `10` и сохраняет цитаты и авторов в `quotes.json`.
+
<p align="center">
<a href="https://dataimpulse.com/?utm_source=scrapling&utm_medium=banner&utm_campaign=scrapling" target="_blank" style="display:flex; justify-content:center; padding:4px 0;">
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/DataImpulse.png" alt="At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies." style="max-height:60px;">
@@ -231,20 +238,20 @@ MySpider().start()
- 🕷️ **Scrapy-подобный Spider API**: Определяйте Spider'ов с `start_urls`, async `parse` callback'ами и объектами `Request`/`Response`.
- ⚡ **Параллельный обход**: Настраиваемые лимиты параллелизма, ограничение скорости по домену и задержки загрузки.
- 🔄 **Поддержка нескольких сессий**: Единый интерфейс для HTTP-запросов и скрытных headless-браузеров в одном Spider - маршрутизируйте запросы к разным сессиям по ID.
-- 💾 **Pause & Resume**: Persistence обхода на основе Checkpoint'ов. Нажмите Ctrl+C для мягкой остановки; перезапустите, чтобы продолжить с того места, где вы остановились.
+- 💾 **Пауза и возобновление**: Сохраняйте ожидающие запросы и посещённые URL с `crawldir`. Остановите процесс через Ctrl+C и запустите с тем же каталогом.
- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени - идеально для UI, конвейеров и длительных обходов.
- 🛡️ **Обнаружение заблокированных запросов**: Автоматическое обнаружение и повторная отправка заблокированных запросов с настраиваемой логикой.
- 🚦 **AutoThrottle**: Больше не нужно угадывать задержки. Spider сам подбирает задержку для каждого домена по скорости ответа сайта, удваивает её (или ждёт столько, сколько просит `Retry-After`), когда сайт начинает блокировать или ограничивать вас, и снова ускоряется, когда это прекращается.
- 🤖 **Соответствие robots.txt**: Опциональный флаг `robots_txt_obey`, который учитывает директивы `Disallow`, `Crawl-delay` и `Request-rate` с кэшированием по доменам.
- 🧪 **Режим разработки**: Кэшируйте ответы на диск при первом запуске и воспроизводите их при последующих запусках - итерируйте над логикой `parse()`, не отправляя повторные запросы к целевым серверам.
-- 🧩 **Готовые шаблоны Spider**: Обойдитесь без шаблонного кода - `CrawlSpider` для перехода по ссылкам на основе правил, `SitemapSpider` для обхода на основе sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` для перебора XML/RSS- и CSV-фидов и `ShopifySpider`, чтобы вытащить все товары любого магазина Shopify через его JSON API, по одному элементу на вариант.
+- 🧩 **Готовые шаблоны Spider**: Обойдитесь без шаблонного кода - `CrawlSpider` для перехода по ссылкам на основе правил, `SitemapSpider` для обхода на основе sitemap/robots.txt, `XMLFeedSpider`/`CSVFeedSpider` для перебора XML/RSS- и CSV-фидов и `ShopifySpider` для сбора товаров из общедоступных магазинов Shopify, по одной записи на вариант.
- 🔗 **Извлечение ссылок**: Отдельный примитив `LinkExtractor` с шаблонами allow/deny, фильтрами доменов, ограничением области через CSS/XPath, фильтрацией расширений и канонизацией - используйте его внутри шаблонов или сам по себе.
- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенные экспортеры JSON/JSONL/CSV/XML с `result.items.to_json()`, `to_jsonl()`, `to_csv()` и `to_xml()`.
### Продвинутая загрузка сайтов с поддержкой Session
- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS fingerprint браузера, заголовки и использовать HTTP/3.
- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `StealthyFetcher`, поддерживающий Chromium от Playwright и Google Chrome.
-- **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher` и подмену fingerprint'ов. Может легко обойти все типы Cloudflare Turnstile/Interstitial с помощью автоматизации.
+- **Обход антибот-защиты**: Продвинутые возможности скрытности с `StealthyFetcher` и подменой отпечатков. Легко обходит любые типы Turnstile/Interstitial от Cloudflare с помощью автоматизации.
- **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession` и `StealthySession` для управления cookie и состоянием между запросами.
- **Ротация Proxy**: Встроенный `ProxyRotator` с циклической или пользовательскими стратегиями для всех типов сессий, а также переопределение Proxy для каждого запроса.
- **Блокировка доменов и рекламы**: Блокируйте запросы к определённым доменам (и их поддоменам) или включите встроенную блокировку рекламы (~3 500 известных рекламных/трекерных доменов) в браузерных Fetcher'ах.
@@ -259,7 +266,7 @@ MySpider().start()
- 🔍 **Поиск похожих элементов**: Автоматически находите элементы, похожие на найденные.
### Возможности для ИИ
-- 🤖 **MCP-сервер**: Позвольте ИИ-чатботам и агентам (Claude/Cursor/и т.д.) скрапить через Scrapling с помощью одноразовых или сессионных инструментов, покрывающих обычные HTTP-запросы (любой метод) и загрузку через скрытый браузер в обход Cloudflare. Страницы сужаются CSS-селекторами и очищаются от prompt-injection-контента до того, как их увидит ИИ, поэтому агент читает меньше, стоит дешевле и не может быть перехвачен скрытым текстом. Скриншоты, удалённые браузеры по CDP и безопасный по умолчанию HTTP-транспорт включены. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP-сервер**: Позвольте ИИ-чатботам и агентам (Claude/Cursor/и др.) скрапить через Scrapling с помощью HTTP-запросов, действий в браузере, скриншотов, сохранённых входов и фоновых ответов API. Наш скрытный браузер и нужные инструменты позволяют подходящим ИИ-моделям обходить любую CAPTCHA, с которой они столкнутся во время работы (не говоря уже о нашем решателе Cloudflare). Использует множество методов, чтобы сократить объём данных для модели, блокировать рекламу и защитить её от Prompt-Injection. Доступны удалённые браузеры через CDP и HTTP-транспорт с настройками аутентификации. ([Демонстрация](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Готовый к установке [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), который обучает кодовых агентов всей библиотеке, чтобы код, который они пишут с Scrapling, соответствовал актуальному API, а не догадкам.
- 📚 **Markdown для RAG**: Превратите любую страницу в чистый, очищенный и готовый для LLM Markdown одной строкой (`page.markdown()`) или обойдите весь сайт, собрав корпус Markdown с помощью шаблона `ContentSpider`, и всё это без участия LLM в процессе. ([документация](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
@@ -306,7 +313,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Дер
data = page.css('#padded_content a').getall()
# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения
-page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()
```
Полная автоматизация браузера
@@ -325,7 +332,7 @@ data = page.css('.quote .text::text').getall()
### Spider'ы
Создавайте полноценные обходчики с параллельными запросами, несколькими типами сессий и Pause & Resume:
```python
-from scrapling.spiders import Spider, Request, Response
+from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
@@ -372,17 +379,18 @@ class MultiSessionSpider(Spider):
```python
QuotesSpider(crawldir="./crawl_data").start()
```
-Нажмите Ctrl+C для мягкой остановки - прогресс сохраняется автоматически. Позже, когда вы снова запустите Spider, передайте тот же `crawldir`, и он продолжит с того места, где остановился.
+Если задан `crawldir`, нажмите Ctrl+C, чтобы сохранить ожидающие запросы и уже посещённые URL. Для продолжения запустите Spider с тем же `crawldir`. Сохраняйте извлечённые данные отдельно.
-Или полностью пропустите написание логики обхода с помощью готовых шаблонов, например, чтобы выгрузить весь каталог магазина Shopify:
+Или совсем не пишите логику обхода: используйте готовые шаблоны, например, чтобы собрать весь каталог магазина Shopify:
```python
from scrapling.spiders import ShopifySpider
class MyStore(ShopifySpider):
target_website = "example.com"
-result = MyStore().start() # Каждый товар в магазине, по одному элементу на вариант
+result = MyStore().start() # Все товары магазина, по одной записи на вариант
```
+Шаблон читает товары из публичных API коллекций, по одной записи на вариант.
### Продвинутый парсинг и навигация
```python
@@ -400,18 +408,19 @@ quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote') # и так далее...
# Найти элемент по текстовому содержимому
-quotes = page.find_by_text('quote', tag='div')
+quotes = page.find_by_text('Albert Einstein', first_match=False)
# Продвинутая навигация
quote_text = page.css('.quote')[0].css('.text::text').get()
-quote_text = page.css('.quote').css('.text::text').getall() # Цепочка селекторов
+quote_texts = page.css('.quote').css('.text::text').getall() # Цепочка селекторов
first_quote = page.css('.quote')[0]
-author = first_quote.next_sibling.css('.author::text')
+author = first_quote.css('.author::text').get()
+next_quote = first_quote.next
parent_container = first_quote.parent
# Связи элементов и подобие
similar_elements = first_quote.find_similar()
-below_elements = first_quote.below_elements()
+below_elements = first_quote.below_elements
```
Вы можете использовать парсер напрямую, если не хотите загружать сайты, как показано ниже:
```python
@@ -426,22 +435,18 @@ page = Selector("<html>...</html>")
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession
-async with FetcherSession(http3=True) as session: # `FetcherSession` контекстно-осведомлён и может работать как в sync, так и в async-режимах
- page1 = session.get('https://quotes.toscrape.com/')
- page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
-
-# Использование async-сессии
-async with AsyncStealthySession(max_pages=2) as session:
- tasks = []
- urls = ['https://example.com/page1', 'https://example.com/page2']
+async def main():
+ async with FetcherSession(http3=True) as http_session: # `FetcherSession` учитывает контекст и работает как с sync, так и с async
+ page1 = await http_session.get('https://quotes.toscrape.com/')
+ page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
+ print(len(page1.css('.quote')), len(page2.css('.quote')))
- for url in urls:
- task = session.fetch(url)
- tasks.append(task)
+ async with AsyncStealthySession(max_pages=2) as browser_session:
+ urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
+ pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
+ print([len(page.css('.quote')) for page in pages])
- print(session.get_pool_stats()) # Опционально - статус пула вкладок браузера (занят/свободен/ошибка)
- results = await asyncio.gather(*tasks)
- print(session.get_pool_stats())
+asyncio.run(main())
```
## CLI и интерактивная Shell
@@ -452,13 +457,14 @@ Scrapling включает мощный интерфейс командной с
Запустить интерактивную Web Scraping Shell
```bash
+pip install "scrapling[shell]"
scrapling shell
```
-Извлечь страницы в файл напрямую без программирования (по умолчанию извлекает содержимое внутри тега `body`). Если выходной файл заканчивается на `.txt`, будет извлечено текстовое содержимое цели. Если заканчивается на `.md`, это будет Markdown-представление HTML-содержимого; если заканчивается на `.html`, это будет само HTML-содержимое.
+Извлекайте страницы прямо в файл без программирования. Выберите содержимое с `--css-selector` или используйте `--ai-targeted` для выбора и очистки содержимого `body`. Используйте `.txt` для текста, `.md` для Markdown или `.html` для HTML.
```bash
-scrapling extract get 'https://example.com' content.md
-scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Все элементы, соответствующие CSS-селектору '#fromSkipToProducts'
-scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
+scrapling extract get 'https://quotes.toscrape.com/' quotes.md
+scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
+scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare
```
@@ -479,7 +485,7 @@ Scrapling не только мощный - он ещё и невероятно
| 4 | PyQuery | 23.98 | ~12x |
| 5 | Selectolax | 197.02 | ~99x |
| 6 | MechanicalSoup | 1545.15 | ~776.5x |
-| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
+| 7 | BS4 with Lxml | 1562.1 | ~785.0x |
| 8 | BS4 with html5lib | 3412.73 | ~1714.9x |
@@ -489,8 +495,8 @@ Scrapling не только мощный - он ещё и невероятно
| Библиотека | Время (мс) | vs Scrapling |
|-------------|:----------:|:------------:|
-| Scrapling | 2.3 | 1.0x |
-| AutoScraper | 12.58 | 5.47x |
+| Scrapling | 2.3 | 1.0x |
+| AutoScraper | 12.58 | 5.47x |
> Все тесты производительности представляют собой средние значения более 100 запусков. См. [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) для методологии.
@@ -543,7 +549,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Помните, что вам нужно установить зависимости браузеров с помощью `scrapling install` после любого из этих дополнений (если вы ещё этого не сделали)
+ Выполните `scrapling install`, если нужны браузерные fetchers. Для HTTP-запросов устанавливать браузер не нужно.
### Docker
Вы также можете установить Docker-образ со всеми дополнениями и браузерами с помощью следующей команды из DockerHub:
@@ -573,7 +579,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
title = {Scrapling},
year = {2024},
url = {https://github.com/D4Vinci/Scrapling},
- note = {An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!}
+ note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
}
```
diff --git a/docs/ai/agent-skill.md b/docs/ai/agent-skill.md
index c108539..6589987 100644
--- a/docs/ai/agent-skill.md
+++ b/docs/ai/agent-skill.md
@@ -1,8 +1,13 @@
+---
+title: "Scrapling Agent Skill: Install and Use"
+description: "Install the Scrapling skill for your coding agent. Give it current API examples and references for fetching, parsing and crawling."
+---
+
# Scrapling Agent Skill
While the [MCP server](mcp-server.md) gives AI chatbots tools to scrape websites for you, the **Scrapling Agent Skill** teaches coding agents the library itself, so the code they write with Scrapling matches the current API instead of guessing from outdated training data.
-The skill follows the [AgentSkills](https://agentskills.io/specification) specification, so it's readable by [Claude Code](https://claude.com/product/claude-code), [OpenClaw](https://github.com/openclaw/openclaw), and other agentic tools. It packages almost all of this documentation website's content as Markdown references that the agent loads on demand, together with ready-to-run examples, so it can answer most Scrapling questions and write correct scraping code without leaving your editor.
+The skill follows the [AgentSkills](https://agentskills.io/specification) specification, so it's readable by [Claude Code](https://claude.com/product/claude-code), [OpenClaw](https://github.com/openclaw/openclaw), and other agentic tools. It packages guides for fetching, parsing, crawls, MCP, and RAG as Markdown references that the agent loads on demand, together with ready-to-run examples, so it can answer most Scrapling questions and write correct scraping code without leaving your editor.
The skill lives in the [agent-skill](https://github.com/D4Vinci/Scrapling/tree/main/agent-skill) directory of the repository under the name `scrapling-official`, and its version always tracks the library version.
@@ -31,7 +36,7 @@ Download the skill's ZIP file [directly from the repository](https://github.com/
## What's inside
- **`SKILL.md`**: The entry point the agent reads first. It covers setup, which fetcher to choose, and the safety rules the agent must follow, like always passing `--ai-targeted` to the CLI commands for prompt injection protection.
-- **`references/`**: Markdown mirrors of the documentation pages (parsing, fetching, spiders, CLI, MCP server, and integrations), loaded on demand so they don't consume the agent's context until needed.
+- **`references/`**: Markdown mirrors of the documentation pages (parsing, fetching, spiders, MCP server, and integrations), loaded on demand so they don't consume the agent's context until needed.
- **`examples/`**: Ready-to-run scripts the agent can copy from.
We tested the skill on OpenClaw and Claude Code. If you face any issues with it, open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) or reach out on our [Discord server](https://discord.gg/EMgGbDceNQ).
diff --git a/docs/ai/building-rag-systems.md b/docs/ai/building-rag-systems.md
index 34a12ab..88951a4 100644
--- a/docs/ai/building-rag-systems.md
+++ b/docs/ai/building-rag-systems.md
@@ -1,3 +1,8 @@
+---
+title: "Crawl Websites into Markdown for RAG with Scrapling"
+description: "Convert page responses to Markdown or use ContentSpider to collect a website. Prepare a corpus for chunking and indexing in a RAG system."
+---
+
# Building RAG Systems with Scrapling
RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page.
@@ -39,7 +44,7 @@ Two arguments control the output:
markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")
```
-Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model.
+Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: inline hidden elements, `aria-hidden="true"` elements, hidden slots, `<template>` tags, HTML comments, and zero-width characters are removed before conversion. The MCP server uses this cleaning with `main_content_only=True`.
## A whole website to a Markdown corpus
@@ -90,7 +95,7 @@ The JSONL output plugs into any embedding pipeline. A minimal example:
```python
import json
-with open("docs.jsonl") as f:
+with open("docs.jsonl", encoding="utf-8") as f:
for line in f:
page = json.loads(line)
for chunk in split_into_chunks(page["markdown"]):
diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md
index 386b788..f4eaeeb 100644
--- a/docs/ai/mcp-server.md
+++ b/docs/ai/mcp-server.md
@@ -1,3 +1,8 @@
+---
+title: "Scrapling MCP Server: Setup and Scraping Workflows"
+description: "Connect an AI client to Scrapling for HTTP and browser scraping. Use sessions, saved login state, actions, focused extraction and API history."
+---
+
# Scrapling MCP Server Guide
<iframe width="560" height="315" src="https://www.youtube.com/embed/qyFk3ZNwOxE?si=3FHzgcYCb66iJ6e3" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
@@ -78,7 +83,7 @@ Set `pierce_shadow=true` on `browser_fetch_once` or each `browser_fetch` call to
- **Browser Impersonation**: Mimic real browsers with TLS fingerprinting, real browser headers matching that version, and more
- **Session Persistence**: Reuse browser sessions across multiple requests for better performance
- **Ad Blocking**: All browser-based tools automatically block requests to ~3,500 known ad and tracker domains, saving tokens and speeding up page loads
-- **Prompt Injection Protection**: Automatic sanitization of hidden content (CSS-hidden elements, aria-hidden, zero-width characters, HTML comments, template tags) that could be used for prompt injection attacks
+- **Prompt Injection Protection**: Automatic sanitization of hidden content (inline hidden elements, aria-hidden, hidden slots, zero-width characters, HTML comments, template tags) that could be used for prompt injection attacks
#### But why use Scrapling MCP Server instead of other available tools?
@@ -142,7 +147,7 @@ As per the [official article](https://modelcontextprotocol.io/quickstart/user),
1. **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
2. **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
-To ensure it's working, use the full path to the `scrapling-mcp` executable. Open the terminal and execute the following command:
+To ensure it's working, use the full path to the `scrapling-mcp` executable:
1. **MacOS**: `which scrapling-mcp`
2. **Windows**: `where scrapling-mcp`
@@ -175,12 +180,12 @@ If you are using the Docker image, then it would be something like
The same logic applies to [Cursor](https://cursor.com/docs/context/mcp), [WindSurf](https://windsurf.com/university/tutorials/configuring-first-mcp-server), and others.
### Claude Code
-Here it's much simpler to do. If you have [Claude Code](https://www.anthropic.com/claude-code) installed, open the terminal and execute the following command:
+Here it's much simpler to do. With [Claude Code](https://www.anthropic.com/claude-code):
```bash
claude mcp add ScraplingServer "/Users/<MyUsername>/.venv/bin/scrapling-mcp"
```
-Same as above, to get Scrapling's executable path, open the terminal and execute the following command:
+To get Scrapling's executable path:
1. **MacOS**: `which scrapling-mcp`
2. **Windows**: `where scrapling-mcp`
@@ -445,15 +450,15 @@ Here is some technical advice for you.
- Increase timeout for slow-loading sites
### 4. Data Quality
-- Use `main_content_only=true` to avoid navigation/ads
+- Use `main_content_only=true` to select and clean the page's body
- Choose an appropriate `extraction_type` for your use case
### 5. Prompt Injection Protection
The MCP server automatically sanitizes scraped content when `main_content_only` is enabled (the default). This strips hidden content that malicious websites could use to inject instructions into the AI's context:
-- **CSS-hidden elements**: `display:none`, `visibility:hidden`, `opacity:0`, `font-size:0`, `height:0`, `width:0`
+- **Inline hidden styles**: `display:none`, `visibility:hidden`, and zero values for opacity, font size, height, width, max-height, and max-width
- **Accessibility-hidden elements**: `aria-hidden="true"`
-- **Template tags**: `<template>` elements
+- **Template tags and hidden slots**: `<template>` and `<slot hidden>` elements
- **HTML comments**: `<!-- ... -->`
- **Zero-width characters**: Invisible unicode characters like zero-width spaces
diff --git a/docs/api-reference/custom-types.md b/docs/api-reference/custom-types.md
index 805e0d4..8ce9e6c 100644
--- a/docs/api-reference/custom-types.md
+++ b/docs/api-reference/custom-types.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "TextHandler and AttributesHandler API Reference"
+description: "Look up Scrapling text, text-list and attribute-mapping helpers, including regex extraction, JSON conversion and value search."
---
# Custom Types API Reference
diff --git a/docs/api-reference/fetchers.md b/docs/api-reference/fetchers.md
index 922bd15..06bd95f 100644
--- a/docs/api-reference/fetchers.md
+++ b/docs/api-reference/fetchers.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "HTTP and Browser Fetchers API Reference"
+description: "Look up Fetcher, AsyncFetcher, StealthyFetcher and session APIs. Check request options, browser settings and return types."
---
# Fetchers Classes
diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md
index 4bcdeb0..fe7ad0c 100644
--- a/docs/api-reference/mcp-server.md
+++ b/docs/api-reference/mcp-server.md
@@ -1,11 +1,11 @@
---
-search:
- exclude: true
+title: "Scrapling MCP Server API Reference"
+description: "Look up MCP server tools, request models, response models and session operations for HTTP and browser scraping."
---
# MCP Server API Reference
-The **Scrapling MCP Server** provides fourteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
+The **Scrapling MCP Server** provides sixteen tools for web scraping and browser interaction through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
You can start the MCP server by running:
diff --git a/docs/api-reference/proxy-rotation.md b/docs/api-reference/proxy-rotation.md
index 61a8fad..03ded79 100644
--- a/docs/api-reference/proxy-rotation.md
+++ b/docs/api-reference/proxy-rotation.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "ProxyRotator API Reference"
+description: "Look up ProxyRotator, proxy selection and rotation strategies. Configure proxies for HTTP and browser sessions."
---
# Proxy Rotation
diff --git a/docs/api-reference/response.md b/docs/api-reference/response.md
index 26f6895..1e42ce8 100644
--- a/docs/api-reference/response.md
+++ b/docs/api-reference/response.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "Scrapling Response API Reference"
+description: "Look up response status, headers, body, metadata, Markdown conversion and follow. Use the inherited Selector methods to extract data."
---
# Response Class
diff --git a/docs/api-reference/selector.md b/docs/api-reference/selector.md
index a4be82b..624496f 100644
--- a/docs/api-reference/selector.md
+++ b/docs/api-reference/selector.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "Selector and Selectors API Reference"
+description: "Look up HTML parsing, selection, extraction, navigation and adaptive tracking methods on Scrapling Selector and Selectors."
---
# Selector Class
diff --git a/docs/api-reference/spiders.md b/docs/api-reference/spiders.md
index 0a69abb..237353a 100644
--- a/docs/api-reference/spiders.md
+++ b/docs/api-reference/spiders.md
@@ -1,6 +1,6 @@
---
-search:
- exclude: true
+title: "Spiders, Requests and Crawl Results API Reference"
+description: "Look up spider lifecycle hooks, request options, session management, crawl statistics and item export methods."
---
# Spider Classes
diff --git a/docs/benchmarks.md b/docs/benchmarks.md
index c1a4df3..81e6b03 100644
--- a/docs/benchmarks.md
+++ b/docs/benchmarks.md
@@ -1,6 +1,11 @@
+---
+title: "Python HTML Parser Benchmarks: Scrapling and Other Libraries"
+description: "See how Scrapling performs in HTML parsing, text extraction and element similarity tests against other popular Python libraries."
+---
+
# Performance Benchmarks
-Scrapling isn't just powerful - it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.
+Scrapling isn't just powerful - it's also blazing fast. The following benchmarks compare Scrapling's parser with other popular libraries.
### Text Extraction Speed Test (5000 nested elements)
@@ -25,4 +30,4 @@ Scrapling's adaptive element finding capabilities significantly outperform alter
| Scrapling | 2.3 | 1.0x |
| AutoScraper | 12.58 | 5.47x |
-> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
+> All benchmarks represent averages of 100 timed runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
diff --git a/docs/cli/extract-commands.md b/docs/cli/extract-commands.md
index fa75b4a..8813e32 100644
--- a/docs/cli/extract-commands.md
+++ b/docs/cli/extract-commands.md
@@ -1,3 +1,8 @@
+---
+title: "Save Web Pages as Markdown, HTML or Text from the CLI"
+description: "Use scrapling extract to save HTTP or browser results. Filter content with CSS, set request options and choose a supported output format."
+---
+
# Scrapling Extract Command Guide
**Web Scraping through the terminal without requiring any programming!**
@@ -24,7 +29,7 @@ The extract command is a set of simple terminal tools that:
!!! tip "AI-Targeted Mode"
- All extract commands support an `--ai-targeted` flag. When enabled, it extracts only the main body content, strips noise tags (script, style, noscript, svg), removes hidden elements that could be used for prompt injection (CSS-hidden, aria-hidden, template tags), strips zero-width unicode characters, and removes HTML comments. For the browser command (`fetch`), it also automatically enables ad blocking. This is ideal when the output is destined for an AI model.
+ All extract commands support an `--ai-targeted` flag. When enabled, it extracts only the main body content, strips noise tags (script, style, noscript, svg), removes hidden elements that could be used for prompt injection (inline hidden elements, aria-hidden, hidden slots, template tags), strips zero-width unicode characters, and removes HTML comments. For the browser command (`fetch`), it also automatically enables ad blocking. This is ideal when the output is destined for an AI model.
## Quick Start
@@ -103,7 +108,7 @@ We will go through each command in detail below.
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
# Add user agent
- scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
+ scrapling extract get "https://api.site.com" data.txt -H "User-Agent: MyBot 1.0"
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
@@ -145,7 +150,7 @@ We will go through each command in detail below.
scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"
# Send JSON data
- scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
+ scrapling extract post "https://api.site.com" response.txt --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract post --help` as follows:
```bash
@@ -185,7 +190,7 @@ We will go through each command in detail below.
scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"
# Send JSON data
- scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
+ scrapling extract put "https://scrapling.requestcatcher.com/put" response.txt --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract put --help` as follows:
```bash
diff --git a/docs/cli/interactive-shell.md b/docs/cli/interactive-shell.md
index d0c99ae..a612973 100644
--- a/docs/cli/interactive-shell.md
+++ b/docs/cli/interactive-shell.md
@@ -1,3 +1,8 @@
+---
+title: "Test Web Scraping in the Scrapling Interactive Shell"
+description: "Fetch pages, try selectors and inspect results in an IPython shell. Convert curl commands, reuse recent responses and test scraping code."
+---
+
# Scrapling Interactive Shell Guide
<script src="https://asciinema.org/a/736339.js" id="asciicast-736339" async data-autoplay="1" data-loop="1" data-cols="225" data-rows="40" data-start-at="00:06" data-speed="1.5" data-theme="tango"></script>
@@ -54,9 +59,9 @@ len(stories) # 30
# Extract specific data
for story in stories[:3]:
-... title = story.text
-... url = story['href']
-... print(f"{title}: {url}")
+ title = story.text
+ url = story['href']
+ print(f"{title}: {url}")
# Try different approaches
titles = page.css('.titleline>a::text') # Direct text extraction
@@ -95,18 +100,18 @@ The shell automatically tracks your requests and pages:
The `pages` command keeps track of the last five pages (it's a `Selectors` object):
```python
- get('https://site1.com')
- get('https://site2.com')
- get('https://site3.com')
+ get('https://site1.com')
+ get('https://site2.com')
+ get('https://site3.com')
- # Access last 5 pages
- len(pages) # `Selectors` object with `page` history -> 3
- pages[0].url # First page in history -> 'https://site1.com'
- pages[-1].url # Most recent page -> 'https://site3.com'
+ # Access last 5 pages
+ len(pages) # `Selectors` object with `page` history -> 3
+ pages[0].url # First page in history -> 'https://site1.com'
+ pages[-1].url # Most recent page -> 'https://site3.com'
- # Work with historical pages
- for i, old_page in enumerate(pages):
- ... print(f"Page {i}: {old_page.url} - {old_page.status}")
+ # Work with historical pages
+ for i, old_page in enumerate(pages):
+ print(f"Page {i}: {old_page.url} - {old_page.status}")
```
## Additional helpful commands
@@ -116,8 +121,8 @@ The shell automatically tracks your requests and pages:
View scraped pages in your browser:
```python
- get('https://quotes.toscrape.com')
- view(page) # Opens the page HTML in your default browser
+get('https://quotes.toscrape.com')
+view(page) # Opens the page HTML in your default browser
```
### Curl Command Integration
@@ -131,24 +136,23 @@ First, you need to copy a request as a curl command like the following:
- **Convert Curl command to Request Object**
```python
- curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \
- ... -X POST \
- ... -H 'Content-Type: application/json' \
- ... -d '{"name": "test", "value": 123}' '''
+ curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \
+ -X POST \
+ -H 'Content-Type: application/json' \
+ -d '{"name": "test", "value": 123}' '''
- request = uncurl(curl_cmd)
- request.method # -> 'post'
- request.url # -> 'https://scrapling.requestcatcher.com/post'
- request.headers # -> {'Content-Type': 'application/json'}
+ request = uncurl(curl_cmd)
+ request.method # -> 'post'
+ request.url # -> 'https://scrapling.requestcatcher.com/post'
+ request.headers # -> {'Content-Type': 'application/json'}
```
- **Execute Curl Command Directly**
```python
- # Convert and execute in one step
- curl2fetcher(curl_cmd)
- page.status # -> 200
- page.json()['json'] # -> {'name': 'test', 'value': 123}
+ # Convert and execute in one step
+ curl2fetcher(curl_cmd)
+ page.status # -> 200
```
### IPython Features
@@ -180,27 +184,27 @@ Here are a few examples generated via AI:
catalog = get('https://shop.example.com/products')
# Find product links
-product_links = catalog.css('.product-link::attr(href)')
+product_links = catalog.css('.product-link::attr(href)').getall()
print(f"Found {len(product_links)} products")
# Sample a few products first
for link in product_links[:3]:
-... product = get(f"https://shop.example.com{link}")
-... name = product.css('.product-name::text').get('')
-... price = product.css('.price::text').get('')
-... print(f"{name}: {price}")
+ product = get(catalog.urljoin(link))
+ name = product.css('.product-name::text').get('')
+ price = product.css('.price::text').get('')
+ print(f"{name}: {price}")
# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
-... products = []
-... for link in product_links:
-... product = session.get(f"https://shop.example.com{link}")
-... products.append({
-... 'name': product.css('.product-name::text').get(''),
-... 'price': product.css('.price::text').get(''),
-... 'url': link
-... })
+ products = []
+ for link in product_links:
+ product = session.get(catalog.urljoin(link))
+ products.append({
+ 'name': product.css('.product-name::text').get(''),
+ 'price': product.css('.price::text').get(''),
+ 'url': link
+ })
```
#### API Integration and Testing
@@ -218,7 +222,7 @@ with FetcherSession() as session:
101
>>> # Test with different data
->>> updated = put(f'https://jsonplaceholder.typicode.com/posts/{new_post.json()["id"]}',
+>>> updated = put('https://jsonplaceholder.typicode.com/posts/1',
... json={'title': 'Updated Title'})
```
diff --git a/docs/cli/overview.md b/docs/cli/overview.md
index dc4346f..f6bcf73 100644
--- a/docs/cli/overview.md
+++ b/docs/cli/overview.md
@@ -1,3 +1,8 @@
+---
+title: "Scrapling CLI: Scrape from the Terminal"
+description: "Install Scrapling terminal tools and choose between extract commands, the interactive shell and the MCP server. Follow the guide for your task."
+---
+
# Command Line Interface
Since v0.3, Scrapling includes a powerful command-line interface that provides three main capabilities:
@@ -23,8 +28,8 @@ This section requires you to install the extra `shell` dependency group, like th
```bash
pip install "scrapling[shell]"
```
-and the installation of the fetchers' dependencies with the following command
+For browser fetching, also install the browser dependencies:
```bash
scrapling install
```
-This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
\ No newline at end of file
+This downloads Chromium and its system dependencies, and updates the TLD list. HTTP requests do not need this step.
\ No newline at end of file
diff --git a/docs/development/adaptive_storage_system.md b/docs/development/adaptive_storage_system.md
index 788ad6e..28c06ad 100644
--- a/docs/development/adaptive_storage_system.md
+++ b/docs/development/adaptive_storage_system.md
@@ -1,3 +1,8 @@
+---
+title: "Write a Custom Storage Backend for Adaptive Parsing"
+description: "Implement save and retrieve for Scrapling adaptive parsing. Write a custom Redis storage backend to save and retrieve element properties."
+---
+
# Writing your retrieval system
Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature.
@@ -65,4 +70,4 @@ class RedisStorage(StorageSystemMixin):
if data:
return orjson.loads(data)
return None
-```
\ No newline at end of file
+```
diff --git a/docs/development/scrapling_custom_types.md b/docs/development/scrapling_custom_types.md
index ef67d27..bbc4e22 100644
--- a/docs/development/scrapling_custom_types.md
+++ b/docs/development/scrapling_custom_types.md
@@ -1,3 +1,8 @@
+---
+title: "Use Scrapling Text and Attribute Types in Your Code"
+description: "Use TextHandler, TextHandlers and AttributesHandler outside the parser. Learn their standard Python base types, JSON helpers and read-only attributes."
+---
+
# Using Scrapling's custom types
> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
@@ -7,13 +12,13 @@
from scrapling.core.custom_types import TextHandler, AttributesHandler
somestring = TextHandler('{}')
-somestring.json() # '{}'
+parsed = somestring.json()
somedict_1 = AttributesHandler({'a': 1})
somedict_2 = AttributesHandler(a=1)
```
Note that `TextHandler` is a subclass of Python's `str`, so all standard operations/methods that work with Python strings will work.
-If you want to check the type in your code, it's better to use Python's built-in `issubclass` function.
+`parsed` is the dictionary `{}`, not the string `'{}'`. To check an instance, use `isinstance(somestring, TextHandler)`. To check a class, use `issubclass(TextHandler, str)`.
The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
diff --git a/docs/donate.md b/docs/donate.md
index dbaab5a..aabeae6 100644
--- a/docs/donate.md
+++ b/docs/donate.md
@@ -1,3 +1,8 @@
+---
+title: "Support Scrapling and Sponsor Its Development"
+description: "Support Scrapling through GitHub Sponsors or Buy Me a Coffee. See the advertising tiers and benefits for project sponsors."
+---
+
# Support and Advertisement
I've been creating all of these projects in my spare time and have invested considerable resources & effort in providing them to the community for free. By becoming a sponsor, you'd be directly funding my coffee reserves, helping me fulfill my responsibilities, and enabling me to continuously update existing projects and potentially create new ones.
diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md
index 242b25a..16938b7 100644
--- a/docs/fetching/choosing.md
+++ b/docs/fetching/choosing.md
@@ -1,3 +1,8 @@
+---
+title: "Choose HTTP Requests or a Browser for Web Scraping"
+description: "Choose Fetcher or StealthyFetcher for your scraping task. Learn when to use direct HTTP, JavaScript rendering and reusable sessions."
+---
+
# Fetchers basics
## Introduction
@@ -88,4 +93,4 @@ Scripts, styles, and hidden/prompt-injection content are always removed before c
!!! note
- Unlike the [Selector](../parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.
\ No newline at end of file
+ Unlike the [Selector](../parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.
diff --git a/docs/fetching/static.md b/docs/fetching/static.md
index 94d9b43..c5e1a2d 100644
--- a/docs/fetching/static.md
+++ b/docs/fetching/static.md
@@ -1,3 +1,8 @@
+---
+title: "Python HTTP Scraping with Fetcher and FetcherSession"
+description: "Send sync or async HTTP requests with browser impersonation, cookies, proxies and reusable sessions. Extract data from the response with Scrapling."
+---
+
# HTTP requests
The `Fetcher` class provides rapid and lightweight HTTP requests using the high-performance `curl_cffi` library with a lot of stealth capabilities.
@@ -23,7 +28,7 @@ All methods for making requests here share some arguments, so let's discuss them
- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
- **follow_redirects**: Controls redirect behavior. **Defaults to `"safe"`**, which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass `True` to follow all redirects without restriction, or `False` to disable redirects entirely.
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
-- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
+- **retries**: The maximum number of attempts for failed requests, including the first attempt. **Defaults to 3**.
- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.
- **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings or a list of them like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear to come from real browsers at the TLS level. If you pass it a list of strings, it will choose a random one with each request. **Defaults to the latest available Chrome version.**
- **http3**: Use HTTP/3 protocol for requests. **Defaults to False**. It might be problematic if used with `impersonate`.
@@ -85,8 +90,7 @@ page = await AsyncFetcher.get('https://example.com')
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
# With parameters
- page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
->>>
+page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
# With headers
page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
# Basic HTTP authentication
@@ -442,4 +446,4 @@ Use other fetchers when:
- Need browser automation.
- Need advanced anti-bot/stealth capabilities.
-- Need JavaScript support or interacting with dynamic content
\ No newline at end of file
+- Need JavaScript support or interacting with dynamic content
diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md
index 344d2ac..c0fa2cc 100644
--- a/docs/fetching/stealthy.md
+++ b/docs/fetching/stealthy.md
@@ -1,3 +1,8 @@
+---
+title: "Scrape JavaScript Websites with StealthyFetcher"
+description: "Render pages, wait for data, reuse login state and read background API responses with Scrapling browser sessions. Configure proxies and challenge handling."
+---
+
# Browser fetching with StealthyFetcher
The `StealthyFetcher` class loads JavaScript pages and automates Chromium or Google Chrome with anti-bot features. It uses [Playwright's API](https://playwright.dev/python/docs/intro) with stealth patches.
@@ -59,7 +64,7 @@ The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed br
StealthyFetcher supports these anti-bot features:
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically with `solve_cloudflare=True`.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
@@ -96,7 +101,7 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
-| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
+| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. Defaults to `False`. | ✔️ |
| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
diff --git a/docs/index.md b/docs/index.md
index 9467418..312c005 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,7 +1,9 @@
+---
+title: "Scrapling: Python Web Scraping Framework"
+description: "Scrapling - Effortless Web Scraping for the Modern Web!"
+---
+
<style>
-.md-typeset h1 {
- display: none;
-}
[data-md-color-scheme="default"] .only-dark { display: none; }
[data-md-color-scheme="slate"] .only-light { display: none; }
</style>
@@ -14,7 +16,9 @@
</a>
</div>
-<h2 align="center"><i>Effortless Web Scraping for the Modern Web</i></h2><br>
+# Scrapling: Python Web Scraping
+
+<p align="center"><i>Effortless Web Scraping for the Modern Web!</i></p>
Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.
@@ -24,10 +28,10 @@ Use it from Python, the terminal, or your AI agent through its MCP server and Ag
## What you can do with Scrapling
-- **Fetch & interact**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
-- **Parse & extract**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
-- **Crawl & export**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
-- **Scrape with AI or from your terminal**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
+- **[Fetch & interact](fetching/choosing.md)**: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
+- **[Parse & extract](parsing/selection.md)**: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
+- **[Crawl & export](spiders/getting-started.md)**: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
+- **[Scrape with AI or from your terminal](ai/mcp-server.md)**: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
Start with a single page:
@@ -39,22 +43,29 @@ quotes = page.css('.quote .text::text').getall()
print(quotes)
```
-Or scale up to full crawls:
+Or scale up with a spider:
```python
from scrapling.spiders import Spider, Response
-class MySpider(Spider):
- name = "demo"
- start_urls = ["https://example.com/"]
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
- async def parse(self, response: Response):
- for item in response.css('.product'):
- yield {"title": item.css('h2::text').get()}
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
-MySpider().start()
+result = QuotesSpider().start()
+print(len(result.items))
+result.items.to_json("quotes.json")
```
+This prints `10` and saves the quotes and authors to `quotes.json`.
+
## Top Sponsors
<style>
@@ -112,13 +123,13 @@ MySpider().start()
- 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.
- ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays.
- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
-- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
+- 💾 **Pause & Resume**: Save pending requests and seen URLs with `crawldir`. Press Ctrl+C to stop gracefully and restart with the same directory.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
- 🚦 **AutoThrottle**: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what `Retry-After` asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
-- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.
+- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, `XMLFeedSpider`/`CSVFeedSpider` for iterating XML/RSS and CSV feeds, and `ShopifySpider` to collect products from public Shopify stores, one item per variant.
- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.
@@ -140,7 +151,7 @@ MySpider().start()
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
### AI Features
-- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA it faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](ai/agent-skill.md) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](ai/building-rag-systems.md))
@@ -222,7 +233,7 @@ pip install scrapling
```bash
pip install "scrapling[all]"
```
- Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
+ Run `scrapling install` when you need browser fetchers. HTTP requests do not need a browser installation.
### Docker
You can also install a Docker image with all extras and browsers with the following command from DockerHub:
@@ -249,4 +260,4 @@ If you like Scrapling and want to support its development:
## License
-This project is licensed under the BSD-3 License. See the [LICENSE](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE) file for details.
\ No newline at end of file
+This project is licensed under the BSD-3 License. See the [LICENSE](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE) file for details.
diff --git a/docs/integrations/scrapy.md b/docs/integrations/scrapy.md
index 1ecf357..a79e1d1 100644
--- a/docs/integrations/scrapy.md
+++ b/docs/integrations/scrapy.md
@@ -1,3 +1,8 @@
+---
+title: "Use Scrapling Parsing in a Scrapy Spider"
+description: "Convert Scrapy responses to Scrapling or decorate a callback. Keep your Scrapy crawler and use Scrapling selectors and adaptive parsing."
+---
+
# Integrating Scrapling into Scrapy
If you have an existing Scrapy project, you don't need to rewrite it to enjoy Scrapling's parsing API. The Scrapy integration converts Scrapy responses to Scrapling [Response](../fetching/choosing.md#response-object) objects right inside your spider callbacks, so Scrapy keeps handling the crawling while Scrapling handles the parsing.
diff --git a/docs/overrides/main.html b/docs/overrides/main.html
index 153f667..8d48552 100644
--- a/docs/overrides/main.html
+++ b/docs/overrides/main.html
@@ -6,7 +6,32 @@
</a>
{% endblock %}
+{% block htmltitle %}
+ {% if page.url == "index.html" and page.meta and page.meta.title %}
+ <title>{{ page.meta.title | e }}</title>
+ {% else %}
+ {{ super() }}
+ {% endif %}
+{% endblock %}
+
{% block extrahead %}
+ {% set share_title = config.site_name %}
+ {% if page.meta and page.meta.title %}
+ {% set share_title = page.meta.title %}
+ {% elif page.title %}
+ {% set share_title = page.title %}
+ {% endif %}
+ {% set share_description = config.site_description %}
+ {% if page.meta and page.meta.description %}
+ {% set share_description = page.meta.description %}
+ {% endif %}
+ <meta property="og:title" content="{{ share_title | e }}" />
+ <meta property="og:description" content="{{ share_description | e }}" />
+ {% if page.canonical_url %}
+ <meta property="og:url" content="{{ page.canonical_url | e }}" />
+ {% endif %}
+ <meta name="twitter:title" content="{{ share_title | e }}" />
+ <meta name="twitter:description" content="{{ share_description | e }}" />
<!-- Open Graph -->
<meta property="og:image" content="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/main_cover.png" />
<meta property="og:image:type" content="image/png" />
@@ -24,4 +49,4 @@
<!-- General -->
<meta name="author" content="Karim Shoair" />
<meta name="theme-color" content="#673ab7" />
-{% endblock %}
\ No newline at end of file
+{% endblock %}
diff --git a/docs/overview.md b/docs/overview.md
index a782484..e2ddf52 100644
--- a/docs/overview.md
+++ b/docs/overview.md
@@ -1,3 +1,10 @@
+---
+title: "Scrapling Overview: Parsing, HTTP Requests and Stealth Browsers"
+description: "Explore Scrapling's parsing and fetching features with CSS, XPath, text search, HTTP requests and stealth browser examples."
+---
+
+# Get Started with Scrapling
+
## Pick Your Path
Not sure where to start? Pick the path that matches what you're trying to do:
@@ -15,8 +22,8 @@ Not sure where to start? Pick the path that matches what you're trying to do:
We will start by quickly reviewing the parsing capabilities. Then we will fetch websites using custom browsers, make requests, and parse the responses.
Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
-```html
-<html>
+```python
+html_doc = """<html>
<head>
<title>Complex Web Page</title>
<style>
@@ -81,10 +88,11 @@ Here's an HTML document generated by ChatGPT that we will be using as an example
}
</script>
</body>
-</html>
+</html>"""
```
Starting with loading raw HTML above like this
```python
+import re
from scrapling.parser import Selector
page = Selector(html_doc)
page # <data='<html><head><title>Complex Web Page</tit...'>
@@ -170,7 +178,7 @@ page.xpath("//*[@id='products']/div/article")
With this, we just scratched the surface of these functions; more advanced options with these selection methods are shown later.
## Accessing elements' data
It's as simple as
-```python
+```pycon
>>> section_element.tag
'section'
>>> print(section_element.attrib)
@@ -210,7 +218,7 @@ It's as simple as
>>> section_element.generate_css_selector
'#products'
>>> section_element.generate_full_css_selector
-'body > main > #products > #products'
+'body > main > #products'
>>> section_element.generate_xpath_selector
"//*[@id='products']"
>>> section_element.generate_full_xpath_selector
@@ -220,7 +228,7 @@ It's as simple as
## Navigation
Using the elements we found above
-```python
+```pycon
>>> section_element.parent
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
>>> section_element.parent.tag
@@ -242,10 +250,10 @@ True
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the one below
```python
for ancestor in section_element.iterancestors():
- # do something with it...
+ print(ancestor.tag)
```
You can search for a specific ancestor of an element that satisfies a function; all you need to do is pass a function that takes a `Selector` object as an argument and returns `True` if the condition is satisfied or `False` otherwise, like below:
-```python
+```pycon
>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
```
@@ -292,11 +300,11 @@ We have you covered if you deal with dynamic websites like most today!
The `StealthyFetcher` class offers many options for fetching and loading web pages using Chromium-based browsers.
```python
from scrapling.fetchers import StealthyFetcher
-page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
+page = StealthyFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote"))) # -> 10
# The async version of fetch
-page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
+page = await StealthyFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True, wait_selector='.quote')
print(len(page.css(".quote"))) # -> 10
```
It's built on top of [Playwright](https://playwright.dev/python/) with stealth patches. Use the bundled Chromium browser, your installed Google Chrome with `real_chrome=True`, or a running browser through `cdp_url`.
@@ -305,7 +313,7 @@ It's built on top of [Playwright](https://playwright.dev/python/) with stealth p
Some of the things it does:
-1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically with `solve_cloudflare=True`.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
@@ -330,4 +338,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
---
-That's Scrapling at a glance. If you want to learn more, continue to the next section.
\ No newline at end of file
+That's Scrapling at a glance. If you want to learn more, continue to the next section.
diff --git a/docs/parsing/adaptive.md b/docs/parsing/adaptive.md
index b2ba6cf..b91867f 100644
--- a/docs/parsing/adaptive.md
+++ b/docs/parsing/adaptive.md
@@ -1,3 +1,8 @@
+---
+title: "Track and Relocate HTML Elements with Adaptive Parsing"
+description: "Save element details and find matches after HTML changes. Learn how to enable adaptive parsing and use save, retrieve and relocate."
+---
+
# Adaptive scraping
!!! success "Prerequisites"
diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md
index d41ff40..25cc34c 100644
--- a/docs/parsing/main_classes.md
+++ b/docs/parsing/main_classes.md
@@ -1,3 +1,8 @@
+---
+title: "Scrapling Selector and Selectors: Data and Navigation"
+description: "Learn how Selector and Selectors store HTML, text and attributes. Extract values, resolve URLs and move through parents, children and siblings."
+---
+
# Parsing main classes
!!! success "Prerequisites"
@@ -29,7 +34,7 @@ In other words, the main page is a [Selector](#selector) object, and the element
### Arguments explained
The most important one is `content`, it's used to pass the HTML code you want to parse, and it accepts the HTML content as `str` or `bytes`.
-Otherwise, you have the arguments `url`, `adaptive`, `storage`, and `storage_args`. All these arguments are settings used with the `adaptive` feature, and they don't make a difference if you are not going to use that feature, so just ignore them for now, and we will explain them in the [adaptive](adaptive.md) feature page.
+The `url` argument sets the page URL used to resolve relative links and separate saved adaptive data by domain. The `adaptive`, `storage`, and `storage_args` arguments are settings used with the [adaptive](adaptive.md) feature, and they don't make a difference if you are not going to use that feature.
Then you have the arguments for parsing adjustments or adjusting/manipulating the HTML content while the library is parsing it:
@@ -344,60 +349,34 @@ Apart from the standard operations on Python lists, such as iteration and slicin
You can do the following:
Execute CSS and XPath selectors directly on the [Selector](#selector) instances it has, while the return types are the same as [Selector](#selector)'s `css` and `xpath` methods. The arguments are similar, except the `adaptive` argument is not available here. This, of course, makes chaining methods very straightforward.
-```python
->>> page.css('.product_pod a')
-[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
-...]
-
->>> page.css('.product_pod').css('a') # Returns the same result
-[<data='<a href="catalogue/a-light-in-the-attic_...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/a-light-in-the-attic_...' parent='<h3><a href="catalogue/a-light-in-the-at...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<div class="image_container"> <a href="c...'>,
- <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
-...]
+```pycon
+>>> page.css('.product h3::text').getall()
+['Product 1', 'Product 2', 'Product 3']
+>>> page.css('.product').css('h3::text').getall()
+['Product 1', 'Product 2', 'Product 3']
```
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
-```python
->>> page.css('.price_color').re(r'[\d\.]+')
-['51.77',
- '53.74',
- '50.10',
- '47.82',
- '54.23',
-...]
-
->>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
-['a-light-in-the-attic_1000',
- 'tipping-the-velvet_999',
- 'soumission_998',
- 'sharp-objects_997',
-...]
+```pycon
+>>> page.css('.price::text').re(r'[\d.]+')
+['10.99', '20.99', '15.99']
+>>> page.css('.product h3::text').re(r'Product (\d+)')
+['1', '2', '3']
```
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
-```python
-# Find all the products with price '53.23'.
->>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
->>> page.css('.product_pod').search(search_function)
-<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
+
+```pycon
+>>> product = page.css('.product').search(lambda p: p.css('.price::text').get() == '$20.99')
+>>> product.css('h3::text').get()
+'Product 2'
```
-You can use the `filter` method, too, which takes a function like the `search` method but returns an `Selectors` instance of all the [Selector](#selector) instances that satisfy the function
-```python
-# Find all products with prices over $50
->>> filtering_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) > 50
->>> page.css('.product_pod').filter(filtering_function)
-[<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
- <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
- <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>,
-...]
+You can use the `filter` method, too, which takes a function like the `search` method but returns a `Selectors` instance of all the [Selector](#selector) instances that satisfy the function
+
+```pycon
+>>> products = page.css('.product').filter(lambda p: float(p.css('.price::text').re_first(r'[\d.]+')) > 15)
+>>> products.css('h3::text').getall()
+['Product 2', 'Product 3']
```
You can safely access the first or last element without worrying about index errors:
```python
@@ -410,11 +389,11 @@ You can safely access the first or last element without worrying about index err
If you are too lazy like me and want to know the number of [Selector](#selector) instances in a [Selectors](#selectors) instance. You can do this:
```python
-page.css('.product_pod').length
+page.css('.product').length
```
which is equivalent to
```python
-len(page.css('.product_pod'))
+len(page.css('.product'))
```
Yup, like JavaScript :)
@@ -438,21 +417,11 @@ First, we start with the `re` and `re_first` methods. These are the same methods
- **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
- ```python
- >>> page.css('.price_color').re(r'[\d\.]+')
- ['51.77',
- '53.74',
- '50.10',
- '47.82',
- '54.23',
- ...]
-
- >>> page.css('.product_pod h3 a::attr(href)').re(r'catalogue/(.*)/index.html')
- ['a-light-in-the-attic_1000',
- 'tipping-the-velvet_999',
- 'soumission_998',
- 'sharp-objects_997',
- ...]
+ ```pycon
+ >>> page.css('.price::text').re(r'[\d.]+')
+ ['10.99', '20.99', '15.99']
+ >>> page.css('.product h3::text').re(r'Product (\d+)')
+ ['1', '2', '3']
```
To explain the other arguments better, we will use a custom string for each example below
```python
@@ -469,7 +438,7 @@ First, we start with the `re` and `re_first` methods. These are the same methods
# Mixing arguments
>>> test_string.re('hi there', clean_match=True, case_sensitive=False)
- ['hi There']
+ ['hi there']
```
Another use of the idea of replacing strings with `TextHandler` everywhere is that a property like `html_content` returns `TextHandler`, so you can do regex on the HTML content if you want:
```python
diff --git a/docs/parsing/selection.md b/docs/parsing/selection.md
index aef8f3b..9916022 100644
--- a/docs/parsing/selection.md
+++ b/docs/parsing/selection.md
@@ -1,5 +1,10 @@
+---
+title: "Extract HTML Data with CSS, XPath, Text and Regex"
+description: "Select HTML elements with CSS, XPath, text and regex. Find similar elements and extract fields with Scrapling code examples."
+---
+
# Querying elements
-Scrapling currently supports parsing HTML pages exclusively, so it doesn't support XML feeds. This decision was made because the adaptive feature won't work with XML, but that might change soon, so stay tuned :)
+The `Selector` class parses HTML. This decision was made because the adaptive feature won't work with XML. For XML, RSS, or Atom feeds, use [XMLFeedSpider](../spiders/generic-templates.md#xmlfeedspider), which uses an XML parser and gives you each feed node directly.
In Scrapling, there are five main ways to find elements:
@@ -78,9 +83,11 @@ title = page.xpath('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
You can nest and chain selectors as you want, given that they return results
```python
page.css('.product')[0].css('h1:contains("Phone")::text').get()
-page.xpath('//*[@class="product"]')[0].xpath('//h1[contains(text(),"Phone")]/text()').get()
+page.xpath('//*[@class="product"]')[0].xpath('.//h1[contains(text(),"Phone")]/text()').get()
page.xpath('//*[@class="product"]')[0].css('h1:contains("Phone")::text').get()
```
+Use `.//` for an XPath relative to the current element. A leading `//` searches the whole document, even after selecting a product.
+
Another example
All links that have 'image' in their 'href' attribute
@@ -236,12 +243,14 @@ Get the `href` attribute from all similar elements
```
To increase the complexity a little bit, let's say we want to get all the books' data using that element as a starting point for some reason
```python
->>> for product in element.parent.parent.find_similar():
+>>> first_product = element.parent.parent
+>>> for product in [first_product, *first_product.find_similar()]:
print({
"name": product.css('h3 a::text').get(),
"price": product.css('.price_color')[0].re_first(r'[\d\.]+'),
"stock": product.css('.availability::text').getall()[-1].clean()
})
+{'name': 'Tipping the Velvet', 'price': '53.74', 'stock': 'In stock'}
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
@@ -259,13 +268,13 @@ def extract_product_grid(page):
)
# Find similar product cards
- products = first_product.find_similar()
+ products = [first_product, *first_product.find_similar()]
return [
{
'name': p.css('h3::text').get(),
'price': p.css('.price::text').re_first(r'\d+\.\d{2}'),
- 'stock': 'In stock' in p.text,
+ 'stock': 'In stock' in p.get_all_text(),
'rating': p.css('.rating')[0].attrib.get('data-rating')
}
for p in products
@@ -278,7 +287,7 @@ def extract_table_data(page):
first_row = page.css('table tbody tr')[0]
# Find similar rows
- rows = first_row.find_similar()
+ rows = [first_row, *first_row.find_similar()]
return [
{
@@ -298,7 +307,7 @@ def extract_form_fields(page):
)
# Find similar field containers
- fields = first_field.find_similar()
+ fields = [first_field, *first_field.find_similar()]
return [
{
@@ -319,7 +328,7 @@ def extract_reviews(page):
)
# Find similar reviews
- all_reviews = review_container.find_similar()
+ all_reviews = [review_container, *review_container.find_similar()]
return [
{
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 582a546..74e38f6 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,8 +1,8 @@
-zensical>=0.0.54
+zensical>=0.0.70
mkdocstrings>=1.0.6
mkdocstrings-python>=2.0.5
griffe-inherited-docstrings>=1.1.3
griffe-runtime-objects>=0.3.1
griffe-sphinx>=0.2.1
black>=26.1.0
-pngquant
\ No newline at end of file
+pngquant
diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md
index 0be6b94..ffa6548 100644
--- a/docs/spiders/advanced.md
+++ b/docs/spiders/advanced.md
@@ -1,3 +1,8 @@
+---
+title: "Crawl Control: AutoThrottle, Checkpoints, Cache and Streaming"
+description: "Control crawl speed and concurrency. Learn response caching, pause and resume, item streaming and the state saved in checkpoints."
+---
+
# Advanced usages
## Introduction
@@ -92,7 +97,7 @@ print(result.stats.autothrottle_delays) # {'example.com': 0.62}
!!! note "Notes:"
- * `download_delay` and any `Crawl-delay` from robots.txt act as a floor. AutoThrottle only ever adjusts the delay above them, so politeness settings are never undercut.<br/>
+ * `download_delay` and robots.txt delays act as a floor, but `autothrottle_max_delay` also caps that floor. Set the maximum at least as high as the configured or robots.txt delay you need to honor.<br/>
* `concurrent_requests_per_domain` does double duty here: it caps how many requests may be in flight to a domain and doubles as AutoThrottle's target, so there's usually no separate target to configure. Setting it is recommended, because while a throttled domain sleeps it holds a slot of the global limiter, and leaving the per-domain limit unlimited means those sleeping slots come out of the budget shared with every other domain.<br/>
* The measured latency is the full fetch, so it includes any internal retries and, for browser sessions, the page render time.<br/>
* `autothrottle_max_delay` caps everything, including `Retry-After`. If a website asks for longer than your ceiling, raise `autothrottle_max_delay` to honor it.<br/>
@@ -142,7 +147,7 @@ The writing to the disk is atomic, so it's totally safe.
!!! tip
- Pressing `Ctrl+C` during a crawl always causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.
+ With `start()`, pressing `Ctrl+C` during a crawl causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.
### Knowing If You're Resuming
@@ -158,7 +163,7 @@ async def on_start(self, resuming: bool = False):
## Development Mode
-When you're iterating on a spider's `parse()` logic, re-hitting the target servers on every run is slow and noisy. Development mode caches every response to disk on the first run and replays them from disk on subsequent runs, so you can tweak your selectors and re-run the spider as many times as you want without making a single network request.
+When you're iterating on a spider's `parse()` logic, re-hitting the target servers on every run is slow and noisy. Development mode caches every response to disk on the first run and replays them from disk on subsequent runs, so you can tweak your selectors and re-run the spider as many times as you want without fetching those pages again.
Enable it by setting `development_mode = True` on your spider:
@@ -172,7 +177,7 @@ class MySpider(Spider):
yield {"title": response.css("title::text").get("")}
```
-The first run fetches normally and stores each response on disk. Every subsequent run serves the same requests from the cache, skipping the network entirely.
+The first run fetches normally and stores each response on disk. Every subsequent run serves the same requests from the cache, skipping the page fetch. With `robots_txt_obey=True`, robots.txt checks run before cache lookup.
### Cache Location
@@ -190,7 +195,7 @@ class MySpider(Spider):
1. **Cache key**: Each response is keyed by the request's fingerprint, so any change to fingerprint-affecting attributes (`fp_include_kwargs`, `fp_include_headers`, `fp_keep_fragments`) will produce a fresh fetch.
2. **Storage format**: One JSON file per response, named `{fingerprint_hex}.json`. The body is base64-encoded so binary content is preserved exactly. Writes are atomic (temp file + rename).
-3. **Replay**: On a cache hit, the engine skips the network entirely, including `download_delay`, rate limiting, and the `is_blocked()` retry path. The cached response goes straight to your callback.
+3. **Replay**: On a cache hit, the engine skips the page fetch, including `download_delay`, rate limiting, and the `is_blocked()` retry path. The cached response goes straight to your callback.
4. **Stats**: Cached requests still count toward `requests_count`, `response_bytes`, and the per-status counters, so your stat output looks the same as a normal crawl. Two extra counters, `cache_hits` and `cache_misses`, let you see how the cache performed.
### Clearing the Cache
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md
index 8895cd2..979e6ef 100644
--- a/docs/spiders/architecture.md
+++ b/docs/spiders/architecture.md
@@ -1,3 +1,8 @@
+---
+title: "How Scrapling Web Crawlers Work"
+description: "See how spiders, requests, sessions, scheduling and the crawl engine fit together. Compare the workflow with Scrapy and choose a starting point."
+---
+
# Spiders architecture
!!! success "Prerequisites"
diff --git a/docs/spiders/generic-templates.md b/docs/spiders/generic-templates.md
index a05aa36..a6b1a78 100644
--- a/docs/spiders/generic-templates.md
+++ b/docs/spiders/generic-templates.md
@@ -1,8 +1,13 @@
+---
+title: "Spider Templates for Links, Sitemaps, Feeds and Markdown"
+description: "Choose CrawlSpider, SitemapSpider, XMLFeedSpider, CSVFeedSpider or ContentSpider. Set rules to collect pages, feed rows or a Markdown corpus."
+---
+
# Generic Spider Templates
Most crawls fall into one of two patterns: "follow links matching this pattern" or "crawl every URL listed in the site's sitemap". Scrapling ships templates for both so you don't have to hand-write the same `parse()` boilerplate every time.
-All templates build on `LinkExtractor`, which pulls URLs out of a `Response` (or filters a single URL via `matches()`). `SitemapSpider` additionally parses sitemap.xml / sitemap_index.xml bodies internally (gzip-compressed or not).
+`CrawlSpider` and `SitemapSpider` build on `LinkExtractor`, which pulls URLs out of a `Response` (or filters a single URL via `matches()`). `SitemapSpider` additionally parses sitemap.xml / sitemap_index.xml bodies internally (gzip-compressed or not).
You can use `LinkExtractor` directly inside any plain `Spider.parse()`. The templates just save you the wiring.
diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.md
index 7f6b523..795a677 100644
--- a/docs/spiders/getting-started.md
+++ b/docs/spiders/getting-started.md
@@ -1,3 +1,8 @@
+---
+title: "Build Your First Python Web Crawler with Scrapling"
+description: "Build a spider that extracts items, follows pagination and exports results. Learn start URLs, callbacks, domain limits and robots.txt settings."
+---
+
# Getting started
## Introduction
@@ -176,7 +181,7 @@ When enabled, the spider will:
1. **Pre-fetch robots.txt** for all domains in `start_urls` before the crawl begins (concurrently).
2. **Check every request** against the domain's robots.txt `Disallow` rules. Disallowed requests are silently dropped and counted in `stats.robots_disallowed_count`.
-3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed.
+3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed. When AutoThrottle is enabled, its maximum delay also caps this value. Set `autothrottle_max_delay` high enough to honor the required delay.
Robots.txt files are fetched using the spider's default session and cached per domain for the entire crawl. Domains discovered mid-crawl (not in `start_urls`) have their robots.txt fetched on the first request to that domain.
diff --git a/docs/spiders/platform-templates.md b/docs/spiders/platform-templates.md
index 8f67f67..752867e 100644
--- a/docs/spiders/platform-templates.md
+++ b/docs/spiders/platform-templates.md
@@ -1,3 +1,8 @@
+---
+title: "Scrape Shopify Product Data with Python"
+description: "Scrape a Shopify store's catalog with ShopifySpider. Set the store URL, collect product variants and inspect each item's fields."
+---
+
# Platform Spider Templates
Generic templates cover crawl *patterns* (follow links, walk sitemaps). Platform templates cover *platforms*: site builders that expose the same machine-readable structure across many independent websites, so the spider already knows where the data lives. You only point it at a domain.
diff --git a/docs/spiders/proxy-blocking.md b/docs/spiders/proxy-blocking.md
index d952317..0d6aeb1 100644
--- a/docs/spiders/proxy-blocking.md
+++ b/docs/spiders/proxy-blocking.md
@@ -1,3 +1,8 @@
+---
+title: "Rotate Proxies and Retry Blocked Crawl Requests"
+description: "Use proxies and rotation strategies in Scrapling spiders. Detect blocked responses, retry requests and switch from HTTP to browser sessions."
+---
+
# Proxy management and handling Blocks
## Introduction
diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.md
index 397d53a..34c2bd5 100644
--- a/docs/spiders/requests-responses.md
+++ b/docs/spiders/requests-responses.md
@@ -1,3 +1,8 @@
+---
+title: "Spider Requests, Callbacks and Response.follow"
+description: "Create crawl requests, pass metadata and follow links with response.follow. Learn callbacks, priorities and request deduplication."
+---
+
# Requests & Responses
!!! success "Prerequisites"
diff --git a/docs/spiders/sessions.md b/docs/spiders/sessions.md
index ec8bdf8..989a3a1 100644
--- a/docs/spiders/sessions.md
+++ b/docs/spiders/sessions.md
@@ -1,3 +1,8 @@
+---
+title: "Use HTTP and Browser Sessions in One Crawler"
+description: "Set up named sessions and route spider requests through HTTP or browser fetchers. Reuse cookies, start sessions on demand and inspect API history."
+---
+
# Spiders sessions
!!! success "Prerequisites"
@@ -166,7 +171,7 @@ async def parse(self, response: Response):
Normally, when you use `FetcherSession`, `Fetcher`, or `AsyncFetcher`, you specify the HTTP method to use with the corresponding method like `.get()` and `.post()`. But while using `FetcherSession` in spiders, you can't do this. By default, the request is an _HTTP GET_ request; if you want to use another HTTP method, you have to pass it to the `method` argument, as in the above example. The reason for this is to unify the `Request` interface across all session types.
-For browser sessions (`AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
+With the `"stealth"` session (`AsyncStealthySession`) registered in the multi-session example above, you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
```python
async def parse(self, response: Response):
@@ -183,7 +188,7 @@ async def parse(self, response: Response):
yield response.follow(
"/dynamic-page",
- sid="browser",
+ sid="stealth",
callback=self.parse_dynamic,
wait_selector="div.loaded",
network_idle=True,
@@ -213,7 +218,7 @@ class ProductSpider(Spider):
next_page = response.css("a.next::attr(href)").get()
if next_page:
- yield Request(next_page)
+ yield response.follow(next_page)
async def parse_product(self, response: Response):
yield {
diff --git a/docs/tutorials/migrating_from_beautifulsoup.md b/docs/tutorials/migrating_from_beautifulsoup.md
index 9abda95..1759710 100644
--- a/docs/tutorials/migrating_from_beautifulsoup.md
+++ b/docs/tutorials/migrating_from_beautifulsoup.md
@@ -1,3 +1,8 @@
+---
+title: "Move from BeautifulSoup to Scrapling"
+description: "Map common BeautifulSoup selection, text, attribute and navigation tasks to Scrapling. Review API differences with practical examples."
+---
+
# Migrating from BeautifulSoup to Scrapling
If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
@@ -14,7 +19,7 @@ You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, w
| Finding a single element | `element = soup.find('div', class_='example')` | `element = page.find('div', class_='example')` |
| Finding multiple elements | `elements = soup.find_all('div', class_='example')` | `elements = page.find_all('div', class_='example')` |
| Finding a single element (Example 2) | `element = soup.find('div', attrs={"class": "example"})` | `element = page.find('div', {"class": "example"})` |
-| Finding a single element (Example 3) | `element = soup.find(re.compile("^b"))` | `element = page.find(re.compile("^b"))`<br/>`element = page.find_by_regex(r"^b")` |
+| Finding a single element (Example 3) | `element = soup.find(re.compile("^b"))` | `element = page.find(lambda e: re.search(r"^b", e.tag) is not None)` |
| Finding a single element (Example 4) | `element = soup.find(lambda e: len(list(e.children)) > 0)` | `element = page.find(lambda e: len(e.children) > 0)` |
| Finding a single element (Example 5) | `element = soup.find(["a", "b"])` | `element = page.find(["a", "b"])` |
| Find element by its text content | `element = soup.find(text="some text")` | `element = page.find_by_text("some text", partial=False)` |
@@ -41,8 +46,9 @@ You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, w
| Get previous sibling of an element | `prev_element = element.previous_sibling` | `prev_element = element.previous` |
| Navigating to children | `children = list(element.children)` | `children = element.children` |
| Get all descendants of an element | `children = list(element.descendants)` | `children = element.below_elements` |
-| Filtering a group of elements that satisfies a condition | `group = soup.find('p', 'story').css.filter('a')` | `group = page.find_all('p', 'story').filter(lambda p: p.tag == 'a')` |
+| Select anchors inside story paragraphs | `group = soup.select('p.story a')` | `group = page.css('p.story a')` |
+A compiled regex passed directly to Scrapling's `find()` or `find_all()` matches element text, not tag names. Use the lambda shown above for tag names, or `page.find_by_regex(r"^b")` for direct text.
¹ **Note:** BS4's `find_previous`/`find_all_previous` searches all preceding elements in document order, while Scrapling's `path` only returns ancestors (the parent chain). These are not exact equivalents, but ancestor search covers the most common use case.
@@ -75,7 +81,7 @@ from scrapling import Fetcher
url = 'https://example.com'
page = Fetcher.get(url)
-links = page.css('a::attr(href)')
+links = page.css('a::attr(href)').getall()
for link in links:
print(link)
```
@@ -91,4 +97,4 @@ As you can see, Scrapling simplifies the process by combining fetching and parsi
The documentation provides more details on Scrapling's features and the complete list of arguments that can be passed to all methods.
-This guide should make your transition from BeautifulSoup to Scrapling smooth and straightforward. Happy scraping!
\ No newline at end of file
+This guide should make your transition from BeautifulSoup to Scrapling smooth and straightforward. Happy scraping!
diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md
index ac3c7b3..250b2b8 100644
--- a/docs/tutorials/replacing_ai.md
+++ b/docs/tutorials/replacing_ai.md
@@ -1,8 +1,13 @@
+---
+title: "A Free Alternative to AI for Robust Web Scraping"
+description: "Scrapling got you covered: tackle changing pages, unstable selectors, anti-bot protection and pagination without an LLM call for every page."
+---
+
# Scrapling: A Free Alternative to AI for Robust Web Scraping
Web scraping has long been a vital tool for data extraction, indexing, and preparing datasets, among other purposes. But experienced users often encounter persistent issues that can hinder effectiveness. Recently, there's been a noticeable shift toward AI-based web scraping, driven by its potential to address these challenges.
-In this article, we will discuss these common issues, why companies are shifting toward that approach, the problems with that approach, and how scrapling solves them for you without the cost of using AI.
+In this article, we will discuss these common issues, why companies are shifting toward that approach, the problems with that approach, and how Scrapling solves them for you without the cost of using AI.
## Common issues and challenging goals
@@ -39,27 +44,27 @@ Unless money is irrelevant to you, you will try to find less expensive approache
Scrapling can handle almost all issues you will face during Web Scraping, and the following updates will cover the rest carefully.
### Solving issue T1: Rapidly changing website structures
-That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html) feature was made. You knew I would talk about it, and here we are :)
+That's why the [adaptive](../parsing/adaptive.md) feature was made. You knew I would talk about it, and here we are :)
While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
-I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html#how-the-adaptive-feature-works).
+I have already explained this in more detail, with many examples. Read more from [here](../parsing/adaptive.md#how-the-adaptive-scraping-feature-works).
### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
In these cases, standard selection methods with CSS/XPath selectors won't be optimal, and that's why Scrapling provides three more methods for Selection:
-1. [Selection by element content](https://scrapling.readthedocs.io/en/latest/parsing/selection.html#text-content-selection): Through text content (`find_by_text`) or regex that matches text content (`find_by_regex`)
-2. [Selecting elements similar to another element](https://scrapling.readthedocs.io/en/latest/parsing/selection.html#finding-similar-elements): You find an element, and we will do the rest!
-3. [Selecting elements by filters](https://scrapling.readthedocs.io/en/latest/parsing/selection.html#filters-based-searching): You specify conditions/filters that this element must fulfill, we find it!
+1. [Selection by element content](../parsing/selection.md#text-content-selection): Through text content (`find_by_text`) or regex that matches text content (`find_by_regex`)
+2. [Selecting elements similar to another element](../parsing/selection.md#finding-similar-elements): You find an element, and we will do the rest!
+3. [Selecting elements by filters](../parsing/selection.md#filters-based-searching): You specify conditions/filters that this element must fulfill, we find it!
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
-Proxies and human-like actions alone may not be enough for protected websites. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) combines browser automation with fingerprint patches and options to handle anti-bot protection. Enable `solve_cloudflare=True` for Cloudflare Turnstile and Interstitial challenges.
+Proxies and human-like actions alone may not be enough for protected websites. [StealthyFetcher](../fetching/stealthy.md) combines browser automation with fingerprint patches and options to handle anti-bot protection. Enable `solve_cloudflare=True` for Cloudflare Turnstile and Interstitial challenges.
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
@@ -81,7 +86,7 @@ Then he said What about cases like this:
So, I updated the code like this
```python
price_element_container = page.find_by_regex(r'[\d,]+', first_match=True).parent # Adjusted the regex for this example
-full_price_data = price_element_container.get_all_text(strip=True) # Returns '$45,000' in this case
+full_price_data = price_element_container.get_all_text(separator="", strip=True) # Returns '$45,000' in this case
```
This was enough for his use case. You can use the first regex, and if it doesn't find anything, use the following regex, and so on. Try to cover the most common patterns first, then the less common ones, and so on.
It will be a bit boring, but it's definitely less expensive than AI.
@@ -89,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations
-This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
+Scrapling currently doesn't have a direct method to automatically identify pagination links for you. Its spiders follow pagination links with `response.follow()`.
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'` - you get the idea.
@@ -98,15 +103,15 @@ For a quick comparison.
| Aspect | Scrapling | AI-Based Tools (e.g., Browse AI, Oxylabs) |
|----------------|----------------------------------------------------------------------------|----------------------------------------------------------------------------|
-| Cost Structure | Likely free or low-cost, no per-use fees | Starts at $19/month (Browse AI) to $49/month (Oxylabs), scales with usage |
+| Cost Structure | Free, with no per-use library fees | Paid plans, with pricing based on the provider and usage |
| Setup Effort | Requires little technical expertise, manual setup | Often no-code, easier for non-technical users |
| Usage options | Through code, terminal, or MCP server. | Often through GUI or API, depending on the option the company is providing |
| Scalability | Depends on user implementation | Built-in support for large-scale, managed services |
| Adaptability | High with features like `adaptive` and the non-selectors selection methods | High, automatic with AI, but costly for frequent changes |
-This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
+See current pricing at [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion
While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
-Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
\ No newline at end of file
+Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
diff --git a/pyproject.toml b/pyproject.toml
index d73020e..9250947 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
version = "0.5"
-description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
+description = "Scrapling is an undetectable, powerful, flexible, high-performance Python framework that makes Web Scraping easy and effortless as it should be!"
readme = {file = "README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
authors = [