Scrapy, a fast high-level web crawling & scraping framework for Python.
Expanded doc for idle signal
Iván de Prado Alonsoe94d3ac173d2
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex ee9d95836..530af1e37 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -269,7 +269,11 @@ spider_idleprevent the spider from being closed.Alternatively, you may raise a :exc:`~scrapy.exceptions.CloseSpider`- exception to provide a custom spider closing reason.+ exception to provide a custom spider closing reason. An+ idle handler is the perfect place to put some code that assesses+ the final spider results and update the final closing reason+ accordingly (e.g. setting it to 'too_few_results' instead of+ 'finished').This signal does not support returning deferreds from its handlers.diff --git a/scrapy/core/engine.py b/scrapy/core/engine.pyindex 81a7a5068..0b34d213d 100644--- a/scrapy/core/engine.py+++ b/scrapy/core/engine.py@@ -326,7 +326,7 @@ class ExecutionEngine:Called when a spider gets idle, i.e. when there are no remaining requests to download or schedule.It can be called multiple times. If a handler for the spider_idle signal raises a DontCloseSpiderexception, the spider is not closed until the next loop and this function is guaranteed to be called- (at least) once again. A handler can raise CloseSpider to provide a custom closing reason+ (at least) once again. A handler can raise CloseSpider to provide a custom closing reason."""assert self.spider is not None # typingexpected_ex = (DontCloseSpider, CloseSpider)