Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

refactor(spiders): Make Robots.txt compliance turned off by default

Karim shoairea2dd7866b9c
diff --git a/scrapling/spiders/spider.py b/scrapling/spiders/spider.py
index 52afcbd..6aaa24f 100644
--- a/scrapling/spiders/spider.py
+++ b/scrapling/spiders/spider.py
@@ -73,7 +73,7 @@ class Spider(ABC):
allowed_domains: Set[str] = set()
# Robots.txt compliance
- robots_txt_obey: bool = True
+ robots_txt_obey: bool = False
# Concurrency settings
concurrent_requests: int = 4