Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

refactor(rag): Renamed SiteToMarkdownSpider to ContentSpider

Karim shoair9cf428038bf4
diff --git a/README.md b/README.md
index 893f1ba..9734efb 100644
--- a/README.md
+++ b/README.md
@@ -261,7 +261,7 @@ MySpider().start()
### AI Features
- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
-- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `SiteToMarkdownSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 681a192..3cd3281 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 17afc10..dcc7426 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -402,7 +402,7 @@ Recorded responses belong to the whole browser session. `search()` and `get()` r
## References
You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
- `references/mcp-server.md` - MCP server tools, persistent session management, remote browsers over CDP, authentication, and capabilities
-- `references/building-rag-systems.md` - Converting pages/websites to LLM-ready Markdown with `Response.markdown()` and `SiteToMarkdownSpider` for RAG pipelines
+- `references/building-rag-systems.md` - Converting pages/websites to LLM-ready Markdown with `Response.markdown()` and `ContentSpider` for RAG pipelines
- `references/parsing` - Everything you need for parsing HTML
- `references/fetching` - Everything you need to fetch websites and session persistence
- `references/spiders` - Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format
diff --git a/agent-skill/Scrapling-Skill/references/building-rag-systems.md b/agent-skill/Scrapling-Skill/references/building-rag-systems.md
index 92653b9..8b56783 100644
--- a/agent-skill/Scrapling-Skill/references/building-rag-systems.md
+++ b/agent-skill/Scrapling-Skill/references/building-rag-systems.md
@@ -43,12 +43,12 @@ Whatever you pass, scripts, styles, and hidden content are always removed before
## A whole website to a Markdown corpus
-The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:
+The `ContentSpider` template crawls a website and converts every page, powered by the [spiders framework](spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:
```python
-from scrapling.spiders import SiteToMarkdownSpider
+from scrapling.spiders import ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/docs/"]
allowed_domains = {"example.com"}
@@ -70,9 +70,9 @@ The template requires `allowed_domains` so the crawl stays bound to the target w
Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):
```python
-from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider
+from scrapling.spiders import CrawlRule, LinkExtractor, ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/"]
allowed_domains = {"example.com"}
diff --git a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
index 50187b2..5823c3e 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md
@@ -191,14 +191,14 @@ class PriceSpider(CSVFeedSpider):
Gzipped feeds are decompressed automatically here as well, as shown above for **XMLFeedSpider**.
-## SiteToMarkdownSpider
+## ContentSpider
-`SiteToMarkdownSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page (`url`, `title`, `markdown` keys) and optionally writing one Markdown file per page:
+`ContentSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page (`url`, `title`, `markdown` keys) and optionally writing one Markdown file per page:
```python
-from scrapling.spiders import SiteToMarkdownSpider
+from scrapling.spiders import ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/docs/"]
allowed_domains = {"example.com"}
diff --git a/docs/README_AR.md b/docs/README_AR.md
index 651d535..f49f393 100644
--- a/docs/README_AR.md
+++ b/docs/README_AR.md
@@ -249,7 +249,7 @@ MySpider().start()
### ميزات الذكاء الاصطناعي
- 🤖 **خادم MCP**: دع روبوتات الدردشة ووكلاء الذكاء الاصطناعي (Claude/Cursor/إلخ) يستخرجون البيانات عبر Scrapling بأدوات تعمل لمرة واحدة أو عبر جلسات، وتغطي طلبات HTTP العادية (بأي طريقة) وجلب الصفحات بالمتصفح الخفي الذي يتجاوز Cloudflare. يتم تضييق الصفحات بمحددات CSS وتنظيفها من محتوى حقن الأوامر قبل أن يراها الذكاء الاصطناعي، فيقرأ الوكيل أقل، ويكلف أقل، ولا يمكن اختطافه بنص مخفي. كما تتوفر لقطات الشاشة، والمتصفحات البعيدة عبر CDP، ونقل HTTP آمن افتراضياً. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: [مهارة جاهزة للتثبيت](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) تُعلّم وكلاء البرمجة المكتبة بالكامل، حتى يكون الكود الذي يكتبونه بـ Scrapling مطابقاً للواجهة الحالية بدلاً من التخمين.
-- 📚 **Markdown جاهز لأنظمة RAG**: حوّل أي صفحة إلى Markdown نظيف ومعقّم وجاهز لنماذج اللغة بسطر واحد (`page.markdown()`)، أو ازحف موقعاً كاملاً إلى مجموعة ملفات Markdown باستخدام القالب `SiteToMarkdownSpider`، كل ذلك بدون استخدام أي نموذج لغوي في العملية. ([الوثائق](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **Markdown جاهز لأنظمة RAG**: حوّل أي صفحة إلى Markdown نظيف ومعقّم وجاهز لنماذج اللغة بسطر واحد (`page.markdown()`)، أو ازحف موقعاً كاملاً إلى مجموعة ملفات Markdown باستخدام القالب `ContentSpider`، كل ذلك بدون استخدام أي نموذج لغوي في العملية. ([الوثائق](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### بنية عالية الأداء ومختبرة ميدانياً
- 🚀 **سريع كالبرق**: أداء محسّن يتفوق على معظم مكتبات Web Scraping في Python.
diff --git a/docs/README_CN.md b/docs/README_CN.md
index f5fd5c0..4574e56 100644
--- a/docs/README_CN.md
+++ b/docs/README_CN.md
@@ -249,7 +249,7 @@ MySpider().start()
### AI 功能
- 🤖 **MCP 服务器**:让 AI 聊天机器人和智能体(Claude/Cursor 等)通过 Scrapling 抓取网页,提供一次性工具和基于会话的工具,覆盖普通 HTTP 请求(任意方法)以及可绕过 Cloudflare 的隐身浏览器抓取。页面在交给 AI 之前会先用 CSS 选择器缩小范围并清除提示注入内容,因此智能体读得更少、花得更少,也不会被隐藏文本劫持。还包含页面截图、通过 CDP 控制远程浏览器,以及默认安全的 HTTP 传输。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:[开箱即用的 Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html),让编码智能体全面掌握本库,使它们用 Scrapling 写出的代码符合当前 API,而不是靠猜测。
-- 📚 **RAG 就绪的 Markdown**:用一行代码(`page.markdown()`)把任意页面转换为干净、经过净化、可直接供 LLM 使用的 Markdown,或使用 `SiteToMarkdownSpider` 模板把整个网站抓取为 Markdown 语料库,整个过程无需调用 LLM。([文档](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **RAG 就绪的 Markdown**:用一行代码(`page.markdown()`)把任意页面转换为干净、经过净化、可直接供 LLM 使用的 Markdown,或使用 `ContentSpider` 模板把整个网站抓取为 Markdown 语料库,整个过程无需调用 LLM。([文档](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### 高性能和经过实战测试的架构
- 🚀 **闪电般快速**:优化性能超越大多数 Python 抓取库。
diff --git a/docs/README_DE.md b/docs/README_DE.md
index 415b67e..6c34491 100644
--- a/docs/README_DE.md
+++ b/docs/README_DE.md
@@ -249,7 +249,7 @@ MySpider().start()
### KI-Funktionen
- 🤖 **MCP-Server**: Lassen Sie KI-Chatbots und -Agenten (Claude/Cursor/etc.) über Scrapling scrapen, mit Einmal- oder sitzungsbasierten Tools für einfache HTTP-Anfragen (jede Methode) und Stealth-Browser-Abrufe, die Cloudflare umgehen. Seiten werden mit CSS-Selektoren eingegrenzt und von Prompt-Injection-Inhalten bereinigt, bevor die KI sie sieht, sodass der Agent weniger liest, weniger kostet und nicht durch versteckten Text gekapert werden kann. Screenshots, Remote-Browser über CDP und ein standardmäßig abgesicherter HTTP-Transport sind enthalten. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Ein installationsfertiger [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), der Coding-Agents die gesamte Bibliothek beibringt, damit der Code, den sie mit Scrapling schreiben, zur aktuellen API passt, statt geraten zu werden.
-- 📚 **RAG-fertiges Markdown**: Wandeln Sie jede Seite mit einer Zeile (`page.markdown()`) in sauberes, bereinigtes, LLM-fertiges Markdown um oder crawlen Sie eine ganze Website mit der Vorlage `SiteToMarkdownSpider` in einen Markdown-Korpus, ganz ohne LLM im Prozess. ([Dokumentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **RAG-fertiges Markdown**: Wandeln Sie jede Seite mit einer Zeile (`page.markdown()`) in sauberes, bereinigtes, LLM-fertiges Markdown um oder crawlen Sie eine ganze Website mit der Vorlage `ContentSpider` in einen Markdown-Korpus, ganz ohne LLM im Prozess. ([Dokumentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### Hochleistungs- und praxiserprobte Architektur
- 🚀 **Blitzschnell**: Optimierte Leistung, die die meisten Python-Scraping-Bibliotheken übertrifft.
diff --git a/docs/README_ES.md b/docs/README_ES.md
index 4681f3d..f121a5a 100644
--- a/docs/README_ES.md
+++ b/docs/README_ES.md
@@ -249,7 +249,7 @@ MySpider().start()
### Características de IA
- 🤖 **Servidor MCP**: Permite que los chatbots y agentes de IA (Claude/Cursor/etc) hagan scraping a través de Scrapling con herramientas de un solo uso o basadas en sesiones que cubren peticiones HTTP simples (cualquier método) y cargas con navegador sigiloso que evitan Cloudflare. Las páginas se acotan con selectores CSS y se limpian de contenido de inyección de prompts antes de que la IA las vea, así que el agente lee menos, cuesta menos y no puede ser secuestrado por texto oculto. Incluye capturas de pantalla, navegadores remotos por CDP y un transporte HTTP seguro por defecto. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) listo para instalar que enseña la biblioteca completa a los agentes de programación, para que el código que escriban con Scrapling coincida con la API actual en vez de adivinarla.
-- 📚 **Markdown listo para RAG**: Convierte cualquier página en Markdown limpio, saneado y listo para LLM con una sola línea (`page.markdown()`), o rastrea un sitio web completo hacia un corpus de Markdown con la plantilla `SiteToMarkdownSpider`, todo sin un LLM en el proceso. ([documentación](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **Markdown listo para RAG**: Convierte cualquier página en Markdown limpio, saneado y listo para LLM con una sola línea (`page.markdown()`), o rastrea un sitio web completo hacia un corpus de Markdown con la plantilla `ContentSpider`, todo sin un LLM en el proceso. ([documentación](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### Arquitectura de Alto Rendimiento y Probada en Batalla
- 🚀 **Ultrarrápido**: Rendimiento optimizado que supera a la mayoría de las bibliotecas de Web Scraping de Python.
diff --git a/docs/README_FR.md b/docs/README_FR.md
index 79e66ca..18f5081 100644
--- a/docs/README_FR.md
+++ b/docs/README_FR.md
@@ -249,7 +249,7 @@ MySpider().start()
### Fonctionnalités IA
- 🤖 **Serveur MCP** : Laissez les chatbots et agents IA (Claude/Cursor/etc.) scraper via Scrapling avec des outils ponctuels ou basés sur des sessions couvrant les requêtes HTTP simples (toute méthode) et les chargements par navigateur furtif qui contournent Cloudflare. Les pages sont réduites avec des sélecteurs CSS et nettoyées du contenu d'injection de prompt avant que l'IA ne les voie, donc l'agent lit moins, coûte moins et ne peut pas être détourné par du texte caché. Captures d'écran, navigateurs distants via CDP et transport HTTP sécurisé par défaut sont inclus. ([vidéo de démonstration](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill** : Un [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) prêt à installer qui enseigne toute la bibliothèque aux agents de codage, pour que le code qu'ils écrivent avec Scrapling corresponde à l'API actuelle au lieu d'être deviné.
-- 📚 **Markdown prêt pour le RAG** : Transformez n'importe quelle page en Markdown propre, assaini et prêt pour les LLM en une ligne (`page.markdown()`), ou crawlez un site entier en un corpus Markdown avec le template `SiteToMarkdownSpider`, le tout sans LLM dans la boucle. ([documentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **Markdown prêt pour le RAG** : Transformez n'importe quelle page en Markdown propre, assaini et prêt pour les LLM en une ligne (`page.markdown()`), ou crawlez un site entier en un corpus Markdown avec le template `ContentSpider`, le tout sans LLM dans la boucle. ([documentation](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### Architecture haute performance et éprouvée
- 🚀 **Ultra rapide** : Performance optimisée surpassant la plupart des bibliothèques de scraping Python.
diff --git a/docs/README_JP.md b/docs/README_JP.md
index 19e1e54..61330a5 100644
--- a/docs/README_JP.md
+++ b/docs/README_JP.md
@@ -249,7 +249,7 @@ MySpider().start()
### AI 機能
- 🤖 **MCP サーバー**:AI チャットボットやエージェント(Claude/Cursor など)が Scrapling を通じてスクレイピングできるようにします。単発ツールとセッションベースのツールがあり、通常の HTTP リクエスト(任意のメソッド)と、Cloudflare を回避するステルスブラウザ取得をカバーします。ページは AI に渡す前に CSS セレクターで絞り込まれ、プロンプトインジェクションの内容が除去されるため、エージェントは読む量もコストも減り、隠しテキストに乗っ取られることもありません。スクリーンショット、CDP 経由のリモートブラウザ、デフォルトで安全な HTTP トランスポートも含まれます。([デモ動画](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**:ライブラリ全体をコーディングエージェントに教える、インストールするだけの [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)。エージェントが Scrapling で書くコードが、推測ではなく現在の API に沿ったものになります。
-- 📚 **RAG 対応の Markdown**:1 行(`page.markdown()`)で任意のページをクリーンでサニタイズ済みの LLM 向け Markdown に変換したり、`SiteToMarkdownSpider` テンプレートでサイト全体を Markdown コーパスとしてクロールしたりできます。処理に LLM は一切使いません。([ドキュメント](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **RAG 対応の Markdown**:1 行(`page.markdown()`)で任意のページをクリーンでサニタイズ済みの LLM 向け Markdown に変換したり、`ContentSpider` テンプレートでサイト全体を Markdown コーパスとしてクロールしたりできます。処理に LLM は一切使いません。([ドキュメント](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### 高性能で実戦テスト済みのアーキテクチャ
- 🚀 **超高速**:ほとんどの Python スクレイピングライブラリを上回る最適化されたパフォーマンス。
diff --git a/docs/README_KR.md b/docs/README_KR.md
index a50293f..f8f8440 100644
--- a/docs/README_KR.md
+++ b/docs/README_KR.md
@@ -249,7 +249,7 @@ MySpider().start()
### AI 기능
- 🤖 **MCP 서버**: AI 챗봇과 에이전트(Claude/Cursor 등)가 Scrapling을 통해 스크래핑하도록 해줍니다. 일회성 도구와 세션 기반 도구가 일반 HTTP 요청(모든 메서드)과 Cloudflare를 우회하는 스텔스 브라우저 가져오기를 모두 지원합니다. 페이지는 AI에 전달되기 전에 CSS 선택자로 범위가 좁혀지고 프롬프트 인젝션 콘텐츠가 제거되므로, 에이전트는 더 적게 읽고 더 적게 비용을 쓰며 숨겨진 텍스트에 탈취당하지 않습니다. 스크린샷, CDP를 통한 원격 브라우저, 기본적으로 안전한 HTTP 전송도 포함됩니다. ([데모 영상](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: 설치만 하면 되는 [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html)이 코딩 에이전트에게 라이브러리 전체를 가르쳐, 에이전트가 Scrapling으로 작성하는 코드가 추측이 아니라 현재 API에 맞도록 해줍니다.
-- 📚 **RAG용 Markdown**: 한 줄(`page.markdown()`)로 어떤 페이지든 깨끗하고 정제된 LLM용 Markdown으로 변환하거나, `SiteToMarkdownSpider` 템플릿으로 웹사이트 전체를 Markdown 코퍼스로 크롤링할 수 있습니다. 과정에 LLM을 전혀 사용하지 않습니다. ([문서](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **RAG용 Markdown**: 한 줄(`page.markdown()`)로 어떤 페이지든 깨끗하고 정제된 LLM용 Markdown으로 변환하거나, `ContentSpider` 템플릿으로 웹사이트 전체를 Markdown 코퍼스로 크롤링할 수 있습니다. 과정에 LLM을 전혀 사용하지 않습니다. ([문서](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### 고성능 & 실전 검증된 아키텍처
- 🚀 **초고속**: 대부분의 Python 스크레이핑 라이브러리를 능가하는 최적화된 성능.
diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.md
index fcd853b..c53c270 100644
--- a/docs/README_PT_BR.md
+++ b/docs/README_PT_BR.md
@@ -251,7 +251,7 @@ MySpider().start()
### Recursos de IA
- 🤖 **Servidor MCP**: Permita que chatbots e agentes de IA (Claude/Cursor/etc) façam scraping através do Scrapling com ferramentas de uso único ou baseadas em sessão, cobrindo requisições HTTP simples (qualquer método) e carregamentos com navegador furtivo que contornam o Cloudflare. As páginas são reduzidas com seletores CSS e limpas de conteúdo de injeção de prompt antes que a IA as veja, então o agente lê menos, custa menos e não pode ser sequestrado por texto oculto. Screenshots, navegadores remotos via CDP e um transporte HTTP seguro por padrão estão incluídos. ([vídeo demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Um [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) pronto para instalar que ensina a biblioteca inteira aos agentes de codificação, para que o código que eles escrevem com Scrapling corresponda à API atual em vez de ser adivinhado.
-- 📚 **Markdown pronto para RAG**: Transforme qualquer página em Markdown limpo, sanitizado e pronto para LLMs com uma linha (`page.markdown()`), ou rastreie um site inteiro em um corpus de Markdown com o template `SiteToMarkdownSpider`, tudo sem nenhum LLM no processo. ([documentação](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **Markdown pronto para RAG**: Transforme qualquer página em Markdown limpo, sanitizado e pronto para LLMs com uma linha (`page.markdown()`), ou rastreie um site inteiro em um corpus de Markdown com o template `ContentSpider`, tudo sem nenhum LLM no processo. ([documentação](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### Arquitetura de Alto Desempenho e Testada em Batalha
- 🚀 **Muito Rápido**: Desempenho otimizado que supera a maioria das bibliotecas Python de scraping.
diff --git a/docs/README_RU.md b/docs/README_RU.md
index 435874e..ef5dbb8 100644
--- a/docs/README_RU.md
+++ b/docs/README_RU.md
@@ -251,7 +251,7 @@ MySpider().start()
### Возможности для ИИ
- 🤖 **MCP-сервер**: Позвольте ИИ-чатботам и агентам (Claude/Cursor/и т.д.) скрапить через Scrapling с помощью одноразовых или сессионных инструментов, покрывающих обычные HTTP-запросы (любой метод) и загрузку через скрытый браузер в обход Cloudflare. Страницы сужаются CSS-селекторами и очищаются от prompt-injection-контента до того, как их увидит ИИ, поэтому агент читает меньше, стоит дешевле и не может быть перехвачен скрытым текстом. Скриншоты, удалённые браузеры по CDP и безопасный по умолчанию HTTP-транспорт включены. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: Готовый к установке [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html), который обучает кодовых агентов всей библиотеке, чтобы код, который они пишут с Scrapling, соответствовал актуальному API, а не догадкам.
-- 📚 **Markdown для RAG**: Превратите любую страницу в чистый, очищенный и готовый для LLM Markdown одной строкой (`page.markdown()`) или обойдите весь сайт, собрав корпус Markdown с помощью шаблона `SiteToMarkdownSpider`, и всё это без участия LLM в процессе. ([документация](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
+- 📚 **Markdown для RAG**: Превратите любую страницу в чистый, очищенный и готовый для LLM Markdown одной строкой (`page.markdown()`) или обойдите весь сайт, собрав корпус Markdown с помощью шаблона `ContentSpider`, и всё это без участия LLM в процессе. ([документация](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
### Высокопроизводительная и проверенная в боях архитектура
- 🚀 **Молниеносная скорость**: Оптимизированная производительность, превосходящая большинство Python-библиотек для скрапинга.
diff --git a/docs/ai/building-rag-systems.md b/docs/ai/building-rag-systems.md
index 7cde702..34a12ab 100644
--- a/docs/ai/building-rag-systems.md
+++ b/docs/ai/building-rag-systems.md
@@ -43,12 +43,12 @@ Whatever you pass, scripts, styles, and hidden content are always removed before
## A whole website to a Markdown corpus
-The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](../spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:
+The `ContentSpider` template crawls a website and converts every page, powered by the [spiders framework](../spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:
```python
-from scrapling.spiders import SiteToMarkdownSpider
+from scrapling.spiders import ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/docs/"]
allowed_domains = {"example.com"}
@@ -70,9 +70,9 @@ The template requires `allowed_domains` so the crawl stays bound to the target w
Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](../spiders/generic-templates.md#crawlspider), override `rules()` with your own [LinkExtractor](../spiders/generic-templates.md#linkextractor-reference) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):
```python
-from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider
+from scrapling.spiders import CrawlRule, LinkExtractor, ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/"]
allowed_domains = {"example.com"}
diff --git a/docs/index.md b/docs/index.md
index 170d3eb..5cace71 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -131,7 +131,7 @@ MySpider().start()
### AI Features
- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method) and stealth browser fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](ai/agent-skill.md) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
-- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `SiteToMarkdownSpider` template, all without an LLM in the loop. ([docs](ai/building-rag-systems.md))
+- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `ContentSpider` template, all without an LLM in the loop. ([docs](ai/building-rag-systems.md))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
diff --git a/docs/spiders/generic-templates.md b/docs/spiders/generic-templates.md
index df18d98..454e104 100644
--- a/docs/spiders/generic-templates.md
+++ b/docs/spiders/generic-templates.md
@@ -192,14 +192,16 @@ class PriceSpider(CSVFeedSpider):
Gzipped feeds are decompressed automatically here as well, as shown above for **XMLFeedSpider**.
-## SiteToMarkdownSpider
+<a id="sitetomarkdownspider"></a>
-`SiteToMarkdownSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page and optionally writing one Markdown file per page:
+## ContentSpider
+
+`ContentSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page and optionally writing one Markdown file per page:
```python
-from scrapling.spiders import SiteToMarkdownSpider
+from scrapling.spiders import ContentSpider
-class DocsSpider(SiteToMarkdownSpider):
+class DocsSpider(ContentSpider):
name = "docs"
start_urls = ["https://example.com/docs/"]
allowed_domains = {"example.com"}
diff --git a/scrapling/spiders/__init__.py b/scrapling/spiders/__init__.py
index af7915c..5f011d0 100644
--- a/scrapling/spiders/__init__.py
+++ b/scrapling/spiders/__init__.py
@@ -12,6 +12,7 @@ from .templates import (
ShopifySpider,
XMLFeedSpider,
CSVFeedSpider,
+ ContentSpider,
SiteToMarkdownSpider,
)
from scrapling.engines.toolbelt.custom import Response
@@ -32,5 +33,6 @@ __all__ = [
"ShopifySpider",
"XMLFeedSpider",
"CSVFeedSpider",
+ "ContentSpider",
"SiteToMarkdownSpider",
]
diff --git a/scrapling/spiders/templates/__init__.py b/scrapling/spiders/templates/__init__.py
index f90d054..b4580b3 100644
--- a/scrapling/spiders/templates/__init__.py
+++ b/scrapling/spiders/templates/__init__.py
@@ -2,7 +2,7 @@ from .crawler import CrawlSpider, CrawlRule
from .sitemap import SitemapSpider
from .shopify import ShopifySpider
from .feed import XMLFeedSpider, CSVFeedSpider
-from .site_to_markdown import SiteToMarkdownSpider
+from .site_to_markdown import ContentSpider, SiteToMarkdownSpider
__all__ = [
"CrawlSpider",
@@ -11,5 +11,6 @@ __all__ = [
"ShopifySpider",
"XMLFeedSpider",
"CSVFeedSpider",
+ "ContentSpider",
"SiteToMarkdownSpider",
]
diff --git a/scrapling/spiders/templates/site_to_markdown.py b/scrapling/spiders/templates/site_to_markdown.py
index d2c55ef..0f4af4f 100644
--- a/scrapling/spiders/templates/site_to_markdown.py
+++ b/scrapling/spiders/templates/site_to_markdown.py
@@ -1,4 +1,4 @@
-"""Site-to-Markdown template spider for building RAG/LLM ingestion pipelines."""
+"""Content template spider for building RAG/LLM ingestion pipelines."""
from re import sub as re_sub
from hashlib import sha1
@@ -24,10 +24,10 @@ if TYPE_CHECKING:
from scrapling.engines.toolbelt.custom import Response
-__all__ = ["SiteToMarkdownSpider"]
+__all__ = ["ContentSpider", "SiteToMarkdownSpider"]
-class SiteToMarkdownSpider(CrawlSpider):
+class ContentSpider(CrawlSpider):
"""A spider that crawls a website and converts every page to clean, LLM-ready Markdown.
Yields one item per page with `url`, `title`, and `markdown` keys, and when `output_dir` is set, it also
@@ -91,3 +91,6 @@ class SiteToMarkdownSpider(CrawlSpider):
name = f"{name}-{sha1(url.encode(), usedforsecurity=False).hexdigest()[:8]}"
self._used_names.add(name)
return name
+
+
+SiteToMarkdownSpider = ContentSpider
diff --git a/tests/spiders/test_site_to_markdown.py b/tests/spiders/test_site_to_markdown.py
index 87b7f85..aa0f082 100644
--- a/tests/spiders/test_site_to_markdown.py
+++ b/tests/spiders/test_site_to_markdown.py
@@ -1,11 +1,11 @@
-"""Tests for `SiteToMarkdownSpider`."""
+"""Tests for `ContentSpider`."""
import pytest
from scrapling.engines.toolbelt.custom import Response
from scrapling.spiders.links import LinkExtractor
from scrapling.spiders.request import Request
-from scrapling.spiders.templates import CrawlRule, SiteToMarkdownSpider
+from scrapling.spiders.templates import CrawlRule, ContentSpider
from scrapling.core._types import AsyncGenerator
HTML = """
@@ -39,8 +39,8 @@ async def _collect(agen: AsyncGenerator) -> list:
return [item async for item in agen]
-def _spider(**attrs) -> SiteToMarkdownSpider:
- class S(SiteToMarkdownSpider):
+def _spider(**attrs) -> ContentSpider:
+ class S(ContentSpider):
name = "s"
start_urls = ["https://example.com/"]
allowed_domains = {"example.com"}
@@ -50,9 +50,9 @@ def _spider(**attrs) -> SiteToMarkdownSpider:
return S()
-class TestSiteToMarkdownSpider:
+class TestContentSpider:
def test_requires_allowed_domains(self):
- class S(SiteToMarkdownSpider):
+ class S(ContentSpider):
name = "s"
start_urls = ["https://example.com/"]