π·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: add the first page of the spiders section
Karim shoair10cd46ee238c
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.mdnew file mode 100644index 0000000..70fba8c--- /dev/null+++ b/docs/spiders/architecture.md@@ -0,0 +1,196 @@+## Introduction++!!! success "Prerequisites"++ 1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand the different fetcher types and when to use each one.+ 2. You've completed or read the [Main classes](../parsing/main_classes.md) page to understand the [Selector](../parsing/main_classes.md#selector) and [Response](../fetching/choosing.md#response-object) classes.++Scrapling's spider system is a Scrapy-inspired async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.++If you're familiar with Scrapy, you'll feel right at home. If not, don't worry β the system is designed to be straightforward.++## Architecture Overview++The diagram below shows how data flows through the spider system when a crawl is running:++```mermaid+graph TB+ subgraph Spider["π·οΈ Spider"]+ direction TB+ SR["start_requests()"]+ Parse["parse() / callbacks"]+ Hooks["Lifecycle hooks<br/><small>on_start Β· on_close Β· on_error<br/>is_blocked Β· on_scraped_item</small>"]+ end++ subgraph Engine["βοΈ Crawler Engine"]+ direction TB+ Loop["Crawl loop"]+ Conc["Concurrency control<br/><small>Global limiter Β· Per-domain limiter<br/>Download delay</small>"]+ BlockDet["Blocked request<br/>detection & retry"]+ end++ subgraph Scheduler["π Scheduler"]+ PQ["Priority queue"]+ Dedup["Fingerprint<br/>deduplication"]+ end++ subgraph SessionMgr["π Session Manager"]+ direction TB+ Router["Request router<br/><small>Routes by session ID</small>"]+ subgraph Sessions["Sessions"]+ direction LR+ HTTP["FetcherSession<br/><small>HTTP requests</small>"]+ Dynamic["AsyncDynamicSession<br/><small>Browser automation</small>"]+ Stealth["AsyncStealthySession<br/><small>Anti-bot bypass</small>"]+ end+ end++ subgraph Output["π¦ Output"]+ Items["ItemList<br/><small>to_json() Β· to_jsonl()</small>"]+ Stream["Streaming<br/><small>async for item in spider.stream()</small>"]+ Stats["CrawlStats"]+ end++ Checkpoint["πΎ Checkpoint<br/><small>Pause/Resume</small>"]++ %% Data flow+ SR -- "1. Initial<br/>Requests" --> Scheduler+ Scheduler -- "2. Next<br/>Request" --> Engine+ Engine -- "3. Fetch" --> SessionMgr+ Router --> Sessions+ SessionMgr -- "4. Response" --> Engine+ Engine -- "5. Response" --> Parse+ Parse -- "6a. New Requests" --> Scheduler+ Parse -- "6b. Items (dict)" --> Output+ Engine -. "Periodic save" .-> Checkpoint+ Checkpoint -. "Resume" .-> Scheduler++ %% Styling+ classDef spiderClass fill:#7c4dff,stroke:#333,color:#fff+ classDef engineClass fill:#00897b,stroke:#333,color:#fff+ classDef schedulerClass fill:#1565c0,stroke:#333,color:#fff+ classDef sessionClass fill:#ef6c00,stroke:#333,color:#fff+ classDef outputClass fill:#2e7d32,stroke:#333,color:#fff+ classDef checkpointClass fill:#6d4c41,stroke:#333,color:#fff++ class Spider spiderClass+ class Engine engineClass+ class Scheduler schedulerClass+ class SessionMgr sessionClass+ class Output outputClass+ class Checkpoint checkpointClass+```++## Data Flow++Here's what happens step by step when you run a spider:++### 1. Spider generates initial Requests++The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic (e.g., POST requests, authentication, different callbacks).++### 2. Requests are scheduled++The **Scheduler** receives requests and places them in a priority queue. Before queuing, each request gets a fingerprint (SHA1 hash of URL + method + body + session ID). Duplicate requests are silently dropped unless `dont_filter=True` is set on the request. Higher-priority requests are dequeued first.++### 3. Engine fetches via Session Manager++The **Crawler Engine** dequeues the next request, respecting concurrency limits (global and per-domain) and download delays. It then hands the request to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID).++A spider can have multiple sessions running simultaneously β for example, a fast `FetcherSession` for simple pages and a `AsyncStealthySession` for protected pages. The Session Manager handles starting, routing, and closing all of them.++### 4. Response comes back++The session fetches the page and returns a `Response` object. The engine records statistics (response bytes, status codes, per-session request counts) and checks for blocked responses.++If the response is blocked (determined by `is_blocked()`), the engine retries the request up to `max_blocked_retries` times, calling `retry_blocked_request()` so you can modify the request before retrying (e.g., switch proxies, add headers).++### 5. Spider processes the Response++The engine passes the `Response` to the spider's callback (default: `parse()`). The callback is an async generator that yields results.++### 6. Results are routed++The callback can yield three types of values:++- **`dict`** β Treated as a scraped item. Passed through `on_scraped_item()` for processing/filtering (return `None` to drop it), then added to the `ItemList` or streamed to the consumer.+- **`Request`** β A follow-up request. Checked against `allowed_domains`, normalized, and sent to the Scheduler for queuing.+- **`None`** β Silently ignored.++The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.++### Checkpointing++If `crawldir` is set, the engine periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off β skipping `start_requests()` and restoring the scheduler state.+++## Components++### Spider++The central class you interact with. You subclass `Spider`, define your `start_urls` and `parse()` method, and optionally configure sessions and override lifecycle hooks.++```python+from scrapling.spiders import Spider, Response, Request++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]++ async def parse(self, response: Response):+ for link in response.css("a::attr(href)").getall():+ yield response.follow(link, callback=self.parse_page)++ async def parse_page(self, response: Response):+ yield {"title": response.css("h1::text").get("")}+```++### Crawler Engine++The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits using `anyio` task groups and `CapacityLimiter`, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly β the `Spider.start()` and `Spider.stream()` methods handle it for you.++### Scheduler++A priority queue with built-in URL deduplication. Requests are fingerprinted based on their URL, HTTP method, body, and session ID. The scheduler supports `snapshot()` and `restore()` for the checkpoint system, allowing the crawl state to be saved and resumed.++### Session Manager++Manages one or more named session instances. Each session is one of:++- **`FetcherSession`** β Async HTTP requests via `curl_cffi` (fastest, no JavaScript)+- **`AsyncDynamicSession`** β Playwright browser automation (JavaScript support)+- **`AsyncStealthySession`** β Stealthy browser with anti-bot bypass (Cloudflare, etc.)++When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started eagerly (default) or lazily (started on first use).++### Checkpoint System++Saves the crawler's state (pending requests + seen URL fingerprints) to a pickle file on disk. Writes are atomic (temp file + rename) to prevent corruption. Checkpoints are saved periodically at a configurable interval and on graceful shutdown. On successful completion (not paused), checkpoint files are cleaned up automatically.++### Output++Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass with per-domain bytes, per-session request counts, status code distribution, and timing information.+++## Comparison with Scrapy++If you're coming from Scrapy, here's how Scrapling's spider system maps:++| Concept | Scrapy | Scrapling |+|---------|--------|-----------|+| Spider definition | `scrapy.Spider` subclass | `scrapling.spiders.Spider` subclass |+| Initial requests | `start_requests()` | `start_requests()` |+| Callbacks | `def parse(self, response)` | `async def parse(self, response)` |+| Following links | `response.follow(url)` | `response.follow(url)` |+| Item output | `yield dict` or `yield Item` | `yield dict` |+| Request scheduling | Scheduler + Dupefilter | Scheduler with built-in deduplication |+| Downloading | Downloader + Middlewares | Session Manager with multi-session support |+| Item processing | Item Pipelines | `on_scraped_item()` hook |+| Blocked detection | Custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` |+| Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute |+| Domain filtering | `allowed_domains` | `allowed_domains` |+| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |+| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` |+| Running | `scrapy crawl spider_name` | `MySpider().start()` |+| Streaming | N/A | `async for item in spider.stream()` |+| Multi-session | N/A | Multiple fetcher sessions per spider |diff --git a/zensical.toml b/zensical.tomlindex 841a462..e0b99f1 100644--- a/zensical.toml+++ b/zensical.toml@@ -30,6 +30,9 @@ nav = [{"Dynamic websites" = "fetching/dynamic.md"},{"Dynamic websites with hard protections" = "fetching/stealthy.md"}]},+ {Spiders = [+ {"Architecture" = "spiders/architecture.md"},+ ]},{"Command Line Interface" = [{Overview = "cli/overview.md"},{"Interactive shell" = "cli/interactive-shell.md"},@@ -111,6 +114,9 @@ toggle.name = "Switch to light mode"#[project.markdown_extensions.mkautodoc][project.markdown_extensions.pymdownx.details][project.markdown_extensions.pymdownx.superfences]+custom_fences = [+ {name = "mermaid", class = "mermaid"}+][project.markdown_extensions.pymdownx.inlinehilite][project.markdown_extensions.pymdownx.snippets][project.markdown_extensions.tables]