Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

style(spiders): Lower the default number of concurrent requests

Karim shoair2eea66dad157
diff --git a/scrapling/spiders/spider.py b/scrapling/spiders/spider.py
index f14174f..c994d2b 100644
--- a/scrapling/spiders/spider.py
+++ b/scrapling/spiders/spider.py
@@ -73,7 +73,7 @@ class Spider(ABC):
allowed_domains: Set[str] = set()
# Concurrency settings
- concurrent_requests: int = 16
+ concurrent_requests: int = 4
concurrent_requests_per_domain: int = 0
download_delay: float = 0.0
max_blocked_retries: int = 3