🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(spiders): Adding easy interface for building RAG systems
Karim shoair6ca35d044856
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex e9a2857..20ee4d0 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.mdindex f6c67a4..d3545fd 100644--- a/agent-skill/Scrapling-Skill/SKILL.md+++ b/agent-skill/Scrapling-Skill/SKILL.md@@ -398,6 +398,7 @@ async with AsyncDynamicSession(capture_xhr=r"https://api\.example\.com/.*") as s## ReferencesYou already had a good glimpse of what the library can do. Use the references below to dig deeper when needed- `references/mcp-server.md` - MCP server tools, persistent session management, remote browsers over CDP, authentication, and capabilities+- `references/building-rag-systems.md` - Converting pages/websites to LLM-ready Markdown with `Response.markdown()` and `SiteToMarkdownSpider` for RAG pipelines- `references/parsing` - Everything you need for parsing HTML- `references/fetching` - Everything you need to fetch websites and session persistence- `references/spiders` - Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like formatdiff --git a/agent-skill/Scrapling-Skill/references/building-rag-systems.md b/agent-skill/Scrapling-Skill/references/building-rag-systems.mdnew file mode 100644index 0000000..92653b9--- /dev/null+++ b/agent-skill/Scrapling-Skill/references/building-rag-systems.md@@ -0,0 +1,104 @@+# Building RAG Systems++RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page.++## Installation++```bash+pip install "scrapling[rag]"++scrapling install+```++The `rag` extra installs the fetchers with Markdown conversion support (the `ai`, `shell`, and `all` extras include it too). The `scrapling install` command downloads the browser dependencies, which you only need for the browser-based fetchers.++## One page to Markdown++Every [Response](fetching/choosing.md) has a `markdown()` method:++```python+from scrapling.fetchers import Fetcher++markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)+```++It works with all fetchers, so pages behind Cloudflare are one line away too:++```python+from scrapling.fetchers import StealthyFetcher++markdown = StealthyFetcher.fetch("https://protected.example.com", solve_cloudflare=True).markdown(main_content_only=True)+```++Two arguments control the output:++- `main_content_only`: Convert only the content inside the page's `<body>` tag.+- `css_selector`: Convert only the elements matching a CSS selector (all matches are concatenated). Use it to extract exactly the part your pipeline needs and save tokens:++```python+markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")+```++Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model.++## A whole website to a Markdown corpus++The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:++```python+from scrapling.spiders import SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"+ max_pages = 200++result = DocsSpider().start()+result.items.to_jsonl("docs.jsonl")+```++Each crawled page becomes one item with `url`, `title`, and `markdown` keys. With `output_dir` set, each page is also written to a Markdown file named after its URL, so the run above gives you both a folder of `.md` files and a `docs.jsonl` ready for ingestion.++The template requires `allowed_domains` so the crawl stays bound to the target website. The options:++- `css_selector` / `main_content_only`: Passed to `markdown()` for every page, with `main_content_only` enabled by default.+- `output_dir`: When set, writes one Markdown file per page.+- `max_pages`: Maximum number of pages to convert. Requests already queued when the cap hits may still be fetched, but they aren't converted. `0` (the default) disables it.++Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):++```python+from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/"]+ allowed_domains = {"example.com"}++ def rules(self):+ return [CrawlRule(LinkExtractor(allow=r"/docs/", deny=[r"/docs/changelog/", r"\?print="]))]+```++`deny` wins over `allow`, and a rule can carry a `priority` or a `process_request` hook as with any **CrawlSpider**.++## Feeding a vector store++The JSONL output plugs into any embedding pipeline. A minimal example:++```python+import json++with open("docs.jsonl") as f:+ for line in f:+ page = json.loads(line)+ for chunk in split_into_chunks(page["markdown"]):+ vector_store.add(text=chunk, metadata={"url": page["url"], "title": page["title"]})+```++Use `css_selector` on the spider to cut boilerplate before chunking instead of cleaning it downstream. The less noise you embed, the better your retrieval.++## Interactive alternatives++For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools, and this agent skill teaches coding agents to write this code themselves.diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.mdindex 1ec0ef8..d3fdf98 100644--- a/agent-skill/Scrapling-Skill/references/fetching/choosing.md+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md@@ -75,4 +75,10 @@ page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is```All fetchers return the `Response` object.+The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:+```python+markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)+```+Scripts, styles, and hidden/prompt-injection content are always removed before conversion, and you can pass `css_selector` to convert specific elements only. It requires the `rag` extra (included in `ai`/`shell`/`all` too). See `../building-rag-systems.md` for the full guide.+**Note:** Unlike the [Selector](parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.\ No newline at end of filediff --git a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.mdindex 31475a4..50187b2 100644--- a/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md+++ b/agent-skill/Scrapling-Skill/references/spiders/generic-templates.md@@ -191,6 +191,24 @@ class PriceSpider(CSVFeedSpider):Gzipped feeds are decompressed automatically here as well, as shown above for **XMLFeedSpider**.+## SiteToMarkdownSpider++`SiteToMarkdownSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page (`url`, `title`, `markdown` keys) and optionally writing one Markdown file per page:++```python+from scrapling.spiders import SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"++result = DocsSpider().start()+```++It requires `allowed_domains` to keep the crawl bound to the target website. Options: `css_selector`/`main_content_only` (passed to `markdown()` per page, `main_content_only` enabled by default), `output_dir`, and `max_pages` (max pages to convert, `0` disables). Every page link inside `allowed_domains` is followed by default; override `rules()` with a `LinkExtractor` (`allow` to narrow the crawl, `deny` to drop URL patterns) to control it. The full guide is in `../building-rag-systems.md`.+## Using `LinkExtractor` directlyYou don't have to use the templates. `LinkExtractor` works inside any plain `Spider`:diff --git a/docs/ai/building-rag-systems.md b/docs/ai/building-rag-systems.mdnew file mode 100644index 0000000..7cde702--- /dev/null+++ b/docs/ai/building-rag-systems.md@@ -0,0 +1,104 @@+# Building RAG Systems with Scrapling++RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page.++## Installation++```bash+pip install "scrapling[rag]"++scrapling install+```++The `rag` extra installs the fetchers with Markdown conversion support (the `ai`, `shell`, and `all` extras include it too). The `scrapling install` command downloads the browser dependencies, which you only need for the browser-based fetchers.++## One page to Markdown++Every [Response](../fetching/choosing.md#response-object) has a `markdown()` method:++```python+from scrapling.fetchers import Fetcher++markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)+```++It works with all fetchers, so pages behind Cloudflare are one line away too:++```python+from scrapling.fetchers import StealthyFetcher++markdown = StealthyFetcher.fetch("https://protected.example.com", solve_cloudflare=True).markdown(main_content_only=True)+```++Two arguments control the output:++- `main_content_only`: Convert only the content inside the page's `<body>` tag.+- `css_selector`: Convert only the elements matching a CSS selector (all matches are concatenated). Use it to extract exactly the part your pipeline needs and save tokens:++```python+markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")+```++Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model.++## A whole website to a Markdown corpus++The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](../spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:++```python+from scrapling.spiders import SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"+ max_pages = 200++result = DocsSpider().start()+result.items.to_jsonl("docs.jsonl")+```++Each crawled page becomes one item with `url`, `title`, and `markdown` keys. With `output_dir` set, each page is also written to a Markdown file named after its URL, so the run above gives you both a folder of `.md` files and a `docs.jsonl` ready for ingestion.++The template requires `allowed_domains` so the crawl stays bound to the target website. The options:++- `css_selector` / `main_content_only`: Passed to `markdown()` for every page, with `main_content_only` enabled by default.+- `output_dir`: When set, writes one Markdown file per page.+- `max_pages`: Maximum number of pages to convert. Requests already queued when the cap hits may still be fetched, but they aren't converted. `0` (the default) disables it.++Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](../spiders/generic-templates.md#crawlspider), override `rules()` with your own [LinkExtractor](../spiders/generic-templates.md#linkextractor-reference) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):++```python+from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/"]+ allowed_domains = {"example.com"}++ def rules(self):+ return [CrawlRule(LinkExtractor(allow=r"/docs/", deny=[r"/docs/changelog/", r"\?print="]))]+```++`deny` wins over `allow`, and a rule can carry a `priority` or a `process_request` hook as with any **CrawlSpider**.++## Feeding a vector store++The JSONL output plugs into any embedding pipeline. A minimal example:++```python+import json++with open("docs.jsonl") as f:+ for line in f:+ page = json.loads(line)+ for chunk in split_into_chunks(page["markdown"]):+ vector_store.add(text=chunk, metadata={"url": page["url"], "title": page["title"]})+```++Use `css_selector` on the spider to cut boilerplate before chunking instead of cleaning it downstream. The less noise you embed, the better your retrieval.++## Interactive alternatives++For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools, and the [Agent skill](agent-skill.md) teaches coding agents to write this code themselves.diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.mdindex b2b4bf2..483aa5b 100644--- a/docs/fetching/choosing.md+++ b/docs/fetching/choosing.md@@ -81,6 +81,12 @@ page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is```All fetchers return the `Response` object.+The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:+```python+markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)+```+Scripts, styles, and hidden/prompt-injection content are always removed before conversion, and you can pass `css_selector` to convert specific elements only. It requires the `rag` extra (included in `ai`/`shell`/`all` too), and the [Building RAG systems](../ai/building-rag-systems.md) page covers it in detail.+!!! noteUnlike the [Selector](../parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.\ No newline at end of filediff --git a/docs/index.md b/docs/index.mdindex 8202581..3ddb304 100644--- a/docs/index.md+++ b/docs/index.md@@ -203,6 +203,10 @@ pip install scrapling```bashpip install "scrapling[ai]"```+ - Install dependencies for ([building RAG systems](ai/building-rag-systems.md)):+ ```bash+ pip install "scrapling[rag]"+ ```- Install shell features (Web Scraping shell and the `extract` command):```bashpip install "scrapling[shell]"diff --git a/docs/spiders/generic-templates.md b/docs/spiders/generic-templates.mdindex 3908552..df18d98 100644--- a/docs/spiders/generic-templates.md+++ b/docs/spiders/generic-templates.md@@ -192,6 +192,24 @@ class PriceSpider(CSVFeedSpider):Gzipped feeds are decompressed automatically here as well, as shown above for **XMLFeedSpider**.+## SiteToMarkdownSpider++`SiteToMarkdownSpider` builds on **CrawlSpider** to crawl a website and convert every page to clean, LLM-ready Markdown, yielding one item per page and optionally writing one Markdown file per page:++```python+from scrapling.spiders import SiteToMarkdownSpider++class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"++result = DocsSpider().start()+```++It requires `allowed_domains` to keep the crawl bound to the target website. The full guide, including all its options and how to feed the output to RAG pipelines, is on the [Building RAG systems](../ai/building-rag-systems.md) page.+## Using `LinkExtractor` directlyYou don't have to use the templates. `LinkExtractor` works inside any plain `Spider`:diff --git a/pyproject.toml b/pyproject.tomlindex 9cc376b..e184a49 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -81,6 +81,10 @@ fetchers = ["anyio>=4.14.0","protego>=0.6.2",]+rag = [+ "markdownify>=1.2.0",+ "scrapling[fetchers]",+]ai = ["mcp>=2.0.0","markdownify>=1.2.0",diff --git a/scrapling/core/shell.py b/scrapling/core/shell.pyindex 762b9de..c9aa2cf 100644--- a/scrapling/core/shell.py+++ b/scrapling/core/shell.py@@ -583,7 +583,12 @@ class Convertor:@classmethoddef _convert_to_markdown(cls, body: TextHandler) -> str:"""Convert HTML content to Markdown"""- from markdownify import markdownify+ try:+ from markdownify import markdownify+ except (ImportError, ModuleNotFoundError) as e:+ raise ModuleNotFoundError(+ 'Markdown conversion requires the "markdownify" package. Install it with: pip install "scrapling[rag]"'+ ) from ereturn markdownify(body)diff --git a/scrapling/engines/toolbelt/custom.py b/scrapling/engines/toolbelt/custom.pyindex 48e3e43..ea86233 100644--- a/scrapling/engines/toolbelt/custom.py+++ b/scrapling/engines/toolbelt/custom.py@@ -85,6 +85,22 @@ class Response(Selector):"""Return the raw body of the response as bytes."""return cast(bytes, cast(Sequence, self._raw_body))+ def markdown(self, css_selector: Optional[str] = None, main_content_only: bool = False) -> str:+ """Convert the response content to clean Markdown.++ Scripts, styles, and hidden/prompt-injection content are always removed before conversion, which is+ the same cleaning the MCP server does. Requires the "markdownify" package (`pip install "scrapling[rag]"`).++ :param css_selector: CSS selector to convert only the matching elements. All matches are concatenated.+ :param main_content_only: Convert only the content inside the `<body>` tag.+ """+ from scrapling.core.shell import Convertor++ page = (cast(Selector, self.css("body").first) or self) if main_content_only else self+ page = Convertor._sanitize_for_ai(Convertor._strip_noise_tags(page))+ pages = [page] if not css_selector else page.css(css_selector)+ return "".join(Convertor._convert_to_markdown(element.html_content) for element in pages)+def follow(self,url: str,diff --git a/scrapling/spiders/__init__.py b/scrapling/spiders/__init__.pyindex f89bb6c..af7915c 100644--- a/scrapling/spiders/__init__.py+++ b/scrapling/spiders/__init__.py@@ -5,7 +5,15 @@ from .engine import CrawlerEnginefrom .session import SessionManagerfrom .spider import Spider, SessionConfigurationErrorfrom .links import LinkExtractor-from .templates import CrawlSpider, SitemapSpider, CrawlRule, ShopifySpider, XMLFeedSpider, CSVFeedSpider+from .templates import (+ CrawlSpider,+ SitemapSpider,+ CrawlRule,+ ShopifySpider,+ XMLFeedSpider,+ CSVFeedSpider,+ SiteToMarkdownSpider,+)from scrapling.engines.toolbelt.custom import Response__all__ = [@@ -24,4 +32,5 @@ __all__ = ["ShopifySpider","XMLFeedSpider","CSVFeedSpider",+ "SiteToMarkdownSpider",]diff --git a/scrapling/spiders/templates/__init__.py b/scrapling/spiders/templates/__init__.pyindex b9bc374..f90d054 100644--- a/scrapling/spiders/templates/__init__.py+++ b/scrapling/spiders/templates/__init__.py@@ -2,6 +2,7 @@ from .crawler import CrawlSpider, CrawlRulefrom .sitemap import SitemapSpiderfrom .shopify import ShopifySpiderfrom .feed import XMLFeedSpider, CSVFeedSpider+from .site_to_markdown import SiteToMarkdownSpider__all__ = ["CrawlSpider",@@ -10,4 +11,5 @@ __all__ = ["ShopifySpider","XMLFeedSpider","CSVFeedSpider",+ "SiteToMarkdownSpider",]diff --git a/scrapling/spiders/templates/site_to_markdown.py b/scrapling/spiders/templates/site_to_markdown.pynew file mode 100644index 0000000..d2c55ef--- /dev/null+++ b/scrapling/spiders/templates/site_to_markdown.py@@ -0,0 +1,93 @@+"""Site-to-Markdown template spider for building RAG/LLM ingestion pipelines."""++from re import sub as re_sub+from hashlib import sha1+from urllib.parse import urlparse++from anyio import Path as AsyncPath++from scrapling.spiders.links import LinkExtractor+from scrapling.spiders.request import Request+from scrapling.spiders.templates.crawler import CrawlRule, CrawlSpider+from scrapling.core._types import (+ TYPE_CHECKING,+ Any,+ AsyncGenerator,+ Dict,+ List,+ Optional,+ Set,+ Union,+)++if TYPE_CHECKING:+ from scrapling.engines.toolbelt.custom import Response+++__all__ = ["SiteToMarkdownSpider"]+++class SiteToMarkdownSpider(CrawlSpider):+ """A spider that crawls a website and converts every page to clean, LLM-ready Markdown.++ Yields one item per page with `url`, `title`, and `markdown` keys, and when `output_dir` is set, it also+ writes each page to a Markdown file named after its URL. Every page link inside `allowed_domains` is followed+ by default; override `rules()` with your own `LinkExtractor` to narrow the crawl or drop URL patterns.++ `allowed_domains` is required so the crawl stays bound to the target website(s).++ :cvar css_selector: CSS selector to convert only the matching elements of each page.+ :cvar main_content_only: Convert only the content inside each page's `<body>` tag. Enabled by default.+ :cvar output_dir: When set, each page is also written to this directory as a Markdown file.+ :cvar max_pages: Maximum number of pages to convert. Requests already queued when the cap hits may still+ be fetched, but they aren't converted or followed. `0` disables the cap.+ """++ css_selector: Optional[str] = None+ main_content_only: bool = True+ output_dir: Optional[str] = None+ max_pages: int = 0++ def __init__(self, *args: Any, **kwargs: Any):+ super().__init__(*args, **kwargs)+ if not self.allowed_domains:+ raise ValueError(+ f"{self.__class__.__name__} must set `allowed_domains` to bound the crawl to the target website(s)"+ )+ self._page_count = 0+ self._used_names: Set[str] = set()++ def rules(self) -> List[CrawlRule]:+ """Follow every page link. Override to narrow the crawl or drop URL patterns."""+ return [CrawlRule(LinkExtractor())]++ async def parse(self, response: "Response") -> AsyncGenerator[Union[Dict[str, Any], Request, None], None]:+ """Yield the page as a Markdown item, then follow its links through the crawl rules."""+ if self.max_pages and self._page_count >= self.max_pages:+ return+ self._page_count += 1+ yield {+ "url": response.url,+ "title": str(response.css("title::text").get() or "").strip(),+ "markdown": response.markdown(self.css_selector, self.main_content_only),+ }+ if not self.max_pages or self._page_count < self.max_pages:+ async for request in super().parse(response):+ yield request++ async def on_scraped_item(self, item: Dict[str, Any]) -> Optional[Dict[str, Any]]:+ """Write the item to a Markdown file inside `output_dir` when it's set."""+ if self.output_dir:+ file = AsyncPath(self.output_dir) / f"{self._filename_for(item['url'])}.md"+ await file.parent.mkdir(parents=True, exist_ok=True)+ await file.write_text(item["markdown"], encoding="utf-8")+ return item++ def _filename_for(self, url: str) -> str:+ """Build a unique filesystem-safe name from the URL, suffixing a hash on collisions."""+ parsed = urlparse(url)+ name = re_sub(r"[^A-Za-z0-9._-]+", "-", f"{parsed.netloc}{parsed.path}".strip("/")).strip("-")[:150] or "index"+ if name in self._used_names:+ name = f"{name}-{sha1(url.encode(), usedforsecurity=False).hexdigest()[:8]}"+ self._used_names.add(name)+ return namediff --git a/tests/fetchers/test_response_markdown.py b/tests/fetchers/test_response_markdown.pynew file mode 100644index 0000000..c725183--- /dev/null+++ b/tests/fetchers/test_response_markdown.py@@ -0,0 +1,65 @@+"""Tests for the public `Response.markdown()` method."""++import sys++import pytest++from scrapling.engines.toolbelt.custom import Response++HTML = """+<html>+ <head><title>Docs Home</title><style>.hint { color: red; }</style></head>+ <body>+ <h1>Title</h1>+ <p>Visible content.</p>+ <p style="display:none">hidden instructions</p>+ <script>console.log("noise")</script>+ <div class="a">First</div>+ <div class="a">Second</div>+ </body>+</html>+"""+++def _make_response(url: str = "https://example.com/") -> Response:+ return Response(+ url=url,+ content=HTML,+ status=200,+ reason="OK",+ cookies={},+ headers={},+ request_headers={},+ )+++class TestResponseMarkdown:+ def test_returns_plain_string_with_markdown(self):+ md = _make_response().markdown()+ assert type(md) is str+ assert "Title\n=====" in md+ assert "Visible content." in md+ assert "Docs Home" in md++ def test_sanitization_is_always_applied(self):+ """Scripts, styles, and hidden elements are stripped even without main_content_only"""+ md = _make_response().markdown()+ assert "hidden instructions" not in md+ assert "console.log" not in md+ assert ".hint" not in md++ def test_main_content_only_scopes_to_body(self):+ md = _make_response().markdown(main_content_only=True)+ assert "Docs Home" not in md+ assert "Visible content." in md++ def test_css_selector_concatenates_all_matches(self):+ md = _make_response().markdown(css_selector=".a")+ assert "First" in md+ assert "Second" in md+ assert "Visible content." not in md++ def test_missing_markdownify_raises_friendly_error(self, monkeypatch):+ monkeypatch.setitem(sys.modules, "markdownify", None)+ with pytest.raises(ModuleNotFoundError, match=r"scrapling\[rag\]"):+ _make_response().markdown()diff --git a/tests/spiders/test_site_to_markdown.py b/tests/spiders/test_site_to_markdown.pynew file mode 100644index 0000000..87b7f85--- /dev/null+++ b/tests/spiders/test_site_to_markdown.py@@ -0,0 +1,146 @@+"""Tests for `SiteToMarkdownSpider`."""++import pytest++from scrapling.engines.toolbelt.custom import Response+from scrapling.spiders.links import LinkExtractor+from scrapling.spiders.request import Request+from scrapling.spiders.templates import CrawlRule, SiteToMarkdownSpider+from scrapling.core._types import AsyncGenerator++HTML = """+<html>+ <head><title> Example Site </title></head>+ <body>+ <h1>Welcome</h1>+ <div class="content"><p>Main content.</p></div>+ <a href="/docs/page-1">page 1</a>+ <a href="/blog/post-1">post 1</a>+ </body>+</html>+"""+++def _make_response(url: str = "https://example.com/") -> Response:+ resp = Response(+ url=url,+ content=HTML,+ status=200,+ reason="OK",+ cookies={},+ headers={},+ request_headers={},+ )+ resp.request = Request(url)+ return resp+++async def _collect(agen: AsyncGenerator) -> list:+ return [item async for item in agen]+++def _spider(**attrs) -> SiteToMarkdownSpider:+ class S(SiteToMarkdownSpider):+ name = "s"+ start_urls = ["https://example.com/"]+ allowed_domains = {"example.com"}++ for name, value in attrs.items():+ setattr(S, name, value)+ return S()+++class TestSiteToMarkdownSpider:+ def test_requires_allowed_domains(self):+ class S(SiteToMarkdownSpider):+ name = "s"+ start_urls = ["https://example.com/"]++ with pytest.raises(ValueError, match="allowed_domains"):+ S()++ @pytest.mark.asyncio+ async def test_parse_yields_item_and_follow_requests(self):+ out = await _collect(_spider().parse(_make_response()))+ items = [o for o in out if isinstance(o, dict)]+ requests = [o for o in out if isinstance(o, Request)]++ assert len(items) == 1+ item = items[0]+ assert item["url"] == "https://example.com/"+ assert item["title"] == "Example Site"+ assert type(item["title"]) is str+ assert type(item["markdown"]) is str+ assert "Welcome\n=======" in item["markdown"]+ assert "Example Site" not in item["markdown"], "main_content_only must exclude the <head>"+ assert [r.url for r in requests] == ["https://example.com/docs/page-1", "https://example.com/blog/post-1"]++ @pytest.mark.asyncio+ async def test_max_pages_caps_converted_pages(self):+ spider = _spider(max_pages=1)+ out = await _collect(spider.parse(_make_response()))+ assert len([o for o in out if isinstance(o, dict)]) == 1+ assert not [o for o in out if isinstance(o, Request)]++ beyond_cap = await _collect(spider.parse(_make_response("https://example.com/docs/page-1")))+ assert beyond_cap == []++ @pytest.mark.asyncio+ async def test_rules_override_narrows_the_crawl(self):+ spider = _spider()+ spider.rules = lambda: [CrawlRule(LinkExtractor(allow=r"/blog/"))] # type: ignore[method-assign]+ out = await _collect(spider.parse(_make_response()))+ assert [r.url for r in out if isinstance(r, Request)] == ["https://example.com/blog/post-1"]++ @pytest.mark.asyncio+ async def test_rules_override_drops_url_patterns(self):+ spider = _spider()+ spider.rules = lambda: [CrawlRule(LinkExtractor(deny=r"/blog/"))] # type: ignore[method-assign]+ out = await _collect(spider.parse(_make_response()))+ assert [r.url for r in out if isinstance(r, Request)] == ["https://example.com/docs/page-1"]++ @pytest.mark.asyncio+ async def test_css_selector_narrows_the_markdown(self):+ out = await _collect(_spider(css_selector=".content").parse(_make_response()))+ markdown = [o for o in out if isinstance(o, dict)][0]["markdown"]+ assert "Main content." in markdown+ assert "Welcome" not in markdown+++class TestSiteToMarkdownOutputDir:+ @pytest.mark.asyncio+ async def test_writes_one_file_per_page(self, tmp_path):+ spider = _spider(output_dir=str(tmp_path))+ item = (await _collect(spider.parse(_make_response())))[0]++ returned = await spider.on_scraped_item(item)+ assert returned is item+ file = tmp_path / "example.com.md"+ assert file.exists()+ assert "Welcome" in file.read_text(encoding="utf-8")++ @pytest.mark.asyncio+ async def test_collisions_get_a_hash_suffix(self, tmp_path):+ spider = _spider(output_dir=str(tmp_path))+ item = (await _collect(spider.parse(_make_response())))[0]++ await spider.on_scraped_item(item)+ await spider.on_scraped_item(item)+ names = {f.name for f in tmp_path.iterdir()}+ assert len(names) == 2+ assert "example.com.md" in names+ suffixed = (names - {"example.com.md"}).pop()+ assert suffixed.startswith("example.com-") and suffixed.endswith(".md")++ def test_filenames_are_slugified_from_the_url(self):+ spider = _spider()+ assert spider._filename_for("https://example.com/") == "example.com"+ assert spider._filename_for("https://example.com/docs/page-1") == "example.com-docs-page-1"++ @pytest.mark.asyncio+ async def test_no_output_dir_writes_nothing(self, tmp_path, monkeypatch):+ monkeypatch.chdir(tmp_path)+ spider = _spider()+ item = (await _collect(spider.parse(_make_response())))[0]+ assert await spider.on_scraped_item(item) is item+ assert not list(tmp_path.iterdir())diff --git a/zensical.toml b/zensical.tomlindex 2553502..a82d7a7 100644--- a/zensical.toml+++ b/zensical.toml@@ -46,7 +46,8 @@ nav = []},{"Using with AI" = [{"MCP server" = "ai/mcp-server.md"},- {"Agent skill" = "ai/agent-skill.md"}+ {"Agent skill" = "ai/agent-skill.md"},+ {"Building RAG systems" = "ai/building-rag-systems.md"}]},{"Integrations and migrations" = [{"Integrating into Scrapy" = "integrations/scrapy.md"},