🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: add a page for spiders advanced usage
Karim shoair7b8fc911e1a3
diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.mdnew file mode 100644index 0000000..1a363b9--- /dev/null+++ b/docs/spiders/advanced.md@@ -0,0 +1,313 @@+# Advanced usages++## Introduction++!!! success "Prerequisites"++ 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.++This page covers the spider system's advanced features: concurrency control, pause/resume, streaming, lifecycle hooks, statistics, and logging.++## Concurrency Control++The spider system uses three class attributes to control how aggressively it crawls:++| Attribute | Default | Description |+|----------------------------------|---------|------------------------------------------------------------------|+| `concurrent_requests` | `4` | Maximum number of requests being processed at the same time |+| `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) |+| `download_delay` | `0.0` | Seconds to wait before each request |++```python+class PoliteSpider(Spider):+ name = "polite"+ start_urls = ["https://example.com"]++ # Be gentle with the server+ concurrent_requests = 4+ concurrent_requests_per_domain = 2+ download_delay = 1.0 # Wait 1 second between requests++ async def parse(self, response: Response):+ yield {"title": response.css("title::text").get("")}+```++When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain.++!!! tip++ The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.++### Using uvloop++The `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available:++```python+result = MySpider().start(use_uvloop=True)+```++This can improve throughput for I/O-heavy crawls. You'll need to install `uvloop` (Linux/macOS) or `winloop` (Windows) separately.++## Pause & Resume++The spider supports graceful pause-and-resume via checkpointing. To enable it, pass a `crawldir` directory to the spider constructor:++```python+spider = MySpider(crawldir="crawl_data/my_spider")+result = spider.start()++if result.paused:+ print("Crawl was paused. Run again to resume.")+else:+ print("Crawl completed!")+```++### How It Works++1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits.+2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks.+3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`.+4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.++**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**++You can change the interval as follows:++```python+# Save checkpoint every 2 minutes+spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)+```++The writing to the disk is atomic, so it's totally safe.++!!! tip++ Pressing `Ctrl+C` during a crawl always causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.++### Knowing If You're Resuming++The `on_start()` hook receives a `resuming` flag:++```python+async def on_start(self, resuming: bool = False):+ if resuming:+ self.logger.info("Resuming from checkpoint!")+ else:+ self.logger.info("Starting fresh crawl")+```++## Streaming++For long-running spiders or applications that need real-time access to scraped items, use the `stream()` method instead of `start()`:++```python+import anyio++async def main():+ spider = MySpider()+ async for item in spider.stream():+ print(f"Got item: {item}")+ # Access real-time stats+ print(f"Items so far: {spider.stats.items_scraped}")+ print(f"Requests made: {spider.stats.requests_count}")++anyio.run(main)+```++Key differences from `start()`:++- `stream()` must be called from an async context+- Items are yielded one by one as they're scraped, not collected into a list+- You can access `spider.stats` during iteration for real-time statistics++!!! abstract++ The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results--statistics)++You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed.++```python+import anyio++async def main():+ spider = MySpider(crawldir="crawl_data/my_spider")+ async for item in spider.stream():+ print(f"Got item: {item}")+ # Access real-time stats+ print(f"Items so far: {spider.stats.items_scraped}")+ print(f"Requests made: {spider.stats.requests_count}")++anyio.run(main)+```+You can also use `spider.pause()` to shut down the spider in the code above. If you used it without enabling the checkpoint system, it will just close the crawl.++## Lifecycle Hooks++The spider provides several hooks you can override to add custom behavior at different stages of the crawl:++### on_start++Called before crawling begins. Use it for setup tasks like loading data or initializing resources:++```python+async def on_start(self, resuming: bool = False):+ self.logger.info("Spider starting up")+ # Load seed URLs from a database, initialize counters, etc.+```++### on_close++Called after crawling finishes (whether completed or paused). Use it for cleanup:++```python+async def on_close(self):+ self.logger.info("Spider shutting down")+ # Close database connections, flush buffers, etc.+```++### on_error++Called when a request fails with an exception. Use it for error tracking or custom recovery logic:++```python+async def on_error(self, request: Request, error: Exception):+ self.logger.error(f"Failed: {request.url} - {error}")+ # Log to error tracker, save failed URL for later, etc.+```++### on_scraped_item++Called for every scraped item before it's added to the results. Return the item (modified or not) to keep it, or return `None` to drop it:++```python+async def on_scraped_item(self, item: dict) -> dict | None:+ # Drop items without a title+ if not item.get("title"):+ return None++ # Modify items (e.g., add timestamps)+ item["scraped_at"] = "2026-01-01"+ return item+```++!!! tip++ This hook can also be used to direct items through your own pipelines and drop them from the spider.++### start_requests++Override `start_requests()` for custom initial request generation instead of using `start_urls`:++```python+async def start_requests(self):+ # POST request to log in first+ yield Request(+ "https://example.com/login",+ method="POST",+ data={"user": "admin", "pass": "secret"},+ callback=self.after_login,+ )++async def after_login(self, response: Response):+ # Now crawl the authenticated pages+ yield response.follow("/dashboard", callback=self.parse)+```++## Results & Statistics++The `CrawlResult` returned by `start()` contains both the scraped items and detailed statistics:++```python+result = MySpider().start()++# Items+print(f"Total items: {len(result.items)}")+result.items.to_json("output.json", indent=True)++# Did the crawl complete?+print(f"Completed: {result.completed}")+print(f"Paused: {result.paused}")++# Statistics+stats = result.stats+print(f"Requests: {stats.requests_count}")+print(f"Failed: {stats.failed_requests_count}")+print(f"Blocked: {stats.blocked_requests_count}")+print(f"Offsite filtered: {stats.offsite_requests_count}")+print(f"Items scraped: {stats.items_scraped}")+print(f"Items dropped: {stats.items_dropped}")+print(f"Response bytes: {stats.response_bytes}")+print(f"Duration: {stats.elapsed_seconds:.1f}s")+print(f"Speed: {stats.requests_per_second:.1f} req/s")+```++### Detailed Stats++The `CrawlStats` object tracks granular information:++```python+stats = result.stats++# Status code distribution+print(stats.response_status_count)+# {'status_200': 150, 'status_404': 3, 'status_403': 1}++# Bytes downloaded per domain+print(stats.domains_response_bytes)+# {'example.com': 1234567, 'api.example.com': 45678}++# Requests per session+print(stats.sessions_requests_count)+# {'http': 120, 'stealth': 34}++# Proxies used during the crawl+print(stats.proxies)+# ['http://proxy1:8080', 'http://proxy2:8080']++# Log level counts+print(stats.log_levels_counter)+# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0}++# Timing information+print(stats.start_time) # Unix timestamp when crawl started+print(stats.end_time) # Unix timestamp when crawl finished+print(stats.download_delay) # The download delay used (seconds)++# Concurrency settings used+print(stats.concurrent_requests) # Global concurrency limit+print(stats.concurrent_requests_per_domain) # Per-domain concurrency limit++# Custom stats (set by your spider code)+print(stats.custom_stats)+# {'login_attempts': 3, 'pages_with_errors': 5}++# Export everything as a dict+print(stats.to_dict())+```++## Logging++The spider has a built-in logger accessible via `self.logger`. It's pre-configured with the spider's name and supports several customization options:++| Attribute | Default | Description |+|-----------------------|--------------------------------------------------------------|----------------------------------------------------|+| `logging_level` | `logging.DEBUG` | Minimum log level |+| `logging_format` | `"[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s"` | Log message format |+| `logging_date_format` | `"%Y-%m-%d %H:%M:%S"` | Date format in log messages |+| `log_file` | `None` | Path to a log file (in addition to console output) |++```python+import logging++class MySpider(Spider):+ name = "my_spider"+ start_urls = ["https://example.com"]+ logging_level = logging.INFO+ log_file = "logs/my_spider.log"++ async def parse(self, response: Response):+ self.logger.info(f"Processing {response.url}")+ yield {"title": response.css("title::text").get("")}+```++The log file directory is created automatically if it doesn't exist. Both console and file output use the same format.\ No newline at end of filediff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.mdindex b61e727..c587af8 100644--- a/docs/spiders/requests-responses.md+++ b/docs/spiders/requests-responses.md@@ -157,11 +157,11 @@ yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboYou can fine-tune what goes into the fingerprint using class attributes on your spider:-| Attribute | Default | Effect |-|----------------------|---------|----------------------------------------------------------------------------------------------------------------|+| Attribute | Default | Effect |+|----------------------|---------|-----------------------------------------------------------------------------------------------------------------|| `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint |-| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints |-| `fp_include_headers` | `False` | Include request headers in the fingerprint |+| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints |+| `fp_include_headers` | `False` | Include request headers in the fingerprint |For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs:diff --git a/zensical.toml b/zensical.tomlindex 084c9d4..fc94fa5 100644--- a/zensical.toml+++ b/zensical.toml@@ -36,6 +36,7 @@ nav = [{"Requests & Responses" = "spiders/requests-responses.md"},{"Sessions" = "spiders/sessions.md"},{"Proxy management & Blocking" = "spiders/proxy-blocking.md"},+ {"Advanced features" = "spiders/advanced.md"}]},{"Command Line Interface" = [{Overview = "cli/overview.md"},