Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(spiders): enable robots.txt compliance by default

Abdullaha134fdb8cce8
diff --git a/scrapling/spiders/spider.py b/scrapling/spiders/spider.py
index 6aaa24f..52afcbd 100644
--- a/scrapling/spiders/spider.py
+++ b/scrapling/spiders/spider.py
@@ -73,7 +73,7 @@ class Spider(ABC):
allowed_domains: Set[str] = set()
# Robots.txt compliance
- robots_txt_obey: bool = False
+ robots_txt_obey: bool = True
# Concurrency settings
concurrent_requests: int = 4