Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.

Pengyu CHENf712513ed709
diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst
index cc02369d4..09cb8ed66 100644
--- a/docs/topics/exceptions.rst
+++ b/docs/topics/exceptions.rst
@@ -39,6 +39,14 @@ For example::
if 'Bandwidth exceeded' in response.body:
raise CloseSpider('bandwidth_exceeded')
+DontCloseSpider
+---------------
+
+.. exception:: DontCloseSpider
+
+This exception can be raised in a :signal:`spider_idle` signal handler to
+prevent the spider from being closed.
+
IgnoreRequest
-------------
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 0306ee4a5..cf1588df8 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -189,14 +189,20 @@ spider_idle
the engine starts closing the spider. After the spider has finished
closing, the :signal:`spider_closed` signal is sent.
- You can, for example, schedule some requests in your :signal:`spider_idle`
- handler to prevent the spider from being closed.
+ You may raise a :exc:`~scrapy.exceptions.DontCloseSpider` exception to
+ prevent the spider from being closed.
This signal does not support returning deferreds from their handlers.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.spiders.Spider` object
+.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
+ **not** guarantee that it can prevent the spider from being closed,
+ although it sometimes can. That's because the spider may still remain idle
+ if all the scheduled requests are rejected by the scheduler (e.g. filtered
+ due to duplication).
+
spider_error
------------