Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: updating pages with the new feature

Karim shoair4c3ff2f0f14b
diff --git a/README.md b/README.md
index d577a00..d6b6e41 100644
--- a/README.md
+++ b/README.md
@@ -214,6 +214,7 @@ MySpider().start()
- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
+- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
### Advanced Websites Fetching with Session Support
diff --git a/docs/index.md b/docs/index.md
index 01d9921..9ff2d04 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -99,6 +99,7 @@ MySpider().start()
- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
+- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
### Advanced Websites Fetching with Session Support
diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md
index c35c2b1..9b1f5b9 100644
--- a/docs/spiders/advanced.md
+++ b/docs/spiders/advanced.md
@@ -17,6 +17,7 @@ The spider system uses three class attributes to control how aggressively it cra
| `concurrent_requests` | `4` | Maximum number of requests being processed at the same time |
| `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) |
| `download_delay` | `0.0` | Seconds to wait before each request |
+| `robots_txt_obey` | `False` | Respect robots.txt rules (Disallow, Crawl-delay, Request-rate) |
```python
class PoliteSpider(Spider):
@@ -234,6 +235,7 @@ print(f"Requests: {stats.requests_count}")
print(f"Failed: {stats.failed_requests_count}")
print(f"Blocked: {stats.blocked_requests_count}")
print(f"Offsite filtered: {stats.offsite_requests_count}")
+print(f"Robots.txt disallowed: {stats.robots_disallowed_count}")
print(f"Items scraped: {stats.items_scraped}")
print(f"Items dropped: {stats.items_dropped}")
print(f"Response bytes: {stats.response_bytes}")
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md
index 4ccfad2..82d388f 100644
--- a/docs/spiders/architecture.md
+++ b/docs/spiders/architecture.md
@@ -19,7 +19,7 @@ Here's what happens step by step when you run a spider without many details:
1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic.
2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first.
-3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID).
+3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. If `robots_txt_obey` is enabled, the engine checks the domain's robots.txt rules before proceeding -- disallowed requests are dropped silently. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID).
4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized.
5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing.
6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.
@@ -91,6 +91,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:
| Blocked detection | Through custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` hooks |
| Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute |
| Domain filtering | `allowed_domains` | `allowed_domains` |
+| Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute |
| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |
| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |
| Running | `scrapy crawl spider_name` | `MySpider().start()` |
diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.md
index e619268..9847c98 100644
--- a/docs/spiders/getting-started.md
+++ b/docs/spiders/getting-started.md
@@ -149,6 +149,31 @@ Subdomains are matched automatically, so setting `allowed_domains = {"example.co
When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.
+## Robots.txt Compliance
+
+Set `robots_txt_obey = True` to make the spider respect robots.txt rules before crawling any domain:
+
+```python
+class PoliteSpider(Spider):
+ name = "polite"
+ start_urls = ["https://example.com"]
+ robots_txt_obey = True
+
+ async def parse(self, response: Response):
+ for link in response.css("a::attr(href)").getall():
+ yield response.follow(link, callback=self.parse)
+```
+
+When enabled, the spider will:
+
+1. **Pre-fetch robots.txt** for all domains in `start_urls` before the crawl begins (concurrently).
+2. **Check every request** against the domain's robots.txt `Disallow` rules. Disallowed requests are silently dropped and counted in `stats.robots_disallowed_count`.
+3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed.
+
+Robots.txt files are fetched using the spider's default session and cached per domain for the entire crawl. Domains discovered mid-crawl (not in `start_urls`) have their robots.txt fetched on the first request to that domain.
+
+**Note:** `robots_txt_obey` is turned off by default to avoid surprising behavior. If you enable it, it does not affect your concurrency settings (`concurrent_requests`, `concurrent_requests_per_domain`) -- only the delay between requests is adjusted.
+
## What's Next
Now that you have the basics, you can explore: