Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

fix(spiders): fix a bug with session stats counter

Karim shoairfbc8be5bcc01
diff --git a/scrapling/spiders/engine.py b/scrapling/spiders/engine.py
index 5819241..1abeb76 100644
--- a/scrapling/spiders/engine.py
+++ b/scrapling/spiders/engine.py
@@ -84,7 +84,7 @@ class CrawlerEngine:
self.stats.proxies.append(dict(request._session_kwargs["proxies"]))
try:
response = await self.session_manager.fetch(request)
- self.stats.increment_requests_count(request.sid)
+ self.stats.increment_requests_count(request.sid or self.session_manager.default_session_id)
self.stats.increment_response_bytes(request.domain, len(response.body))
self.stats.increment_status(response.status)