🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
refactor(spiders): Make Robots.txt compliance turned off by default
Karim shoairea2dd7866b9c
diff --git a/scrapling/spiders/spider.py b/scrapling/spiders/spider.pyindex 52afcbd..6aaa24f 100644--- a/scrapling/spiders/spider.py+++ b/scrapling/spiders/spider.py@@ -73,7 +73,7 @@ class Spider(ABC):allowed_domains: Set[str] = set()# Robots.txt compliance- robots_txt_obey: bool = True+ robots_txt_obey: bool = False# Concurrency settingsconcurrent_requests: int = 4