Scrapy, a fast high-level web crawling & scraping framework for Python.
Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
Pengyu CHENf712513ed709
diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rstindex cc02369d4..09cb8ed66 100644--- a/docs/topics/exceptions.rst+++ b/docs/topics/exceptions.rst@@ -39,6 +39,14 @@ For example::if 'Bandwidth exceeded' in response.body:raise CloseSpider('bandwidth_exceeded')+DontCloseSpider+---------------++.. exception:: DontCloseSpider++This exception can be raised in a :signal:`spider_idle` signal handler to+prevent the spider from being closed.+IgnoreRequest-------------diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 0306ee4a5..cf1588df8 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -189,14 +189,20 @@ spider_idlethe engine starts closing the spider. After the spider has finishedclosing, the :signal:`spider_closed` signal is sent.- You can, for example, schedule some requests in your :signal:`spider_idle`- handler to prevent the spider from being closed.+ You may raise a :exc:`~scrapy.exceptions.DontCloseSpider` exception to+ prevent the spider from being closed.This signal does not support returning deferreds from their handlers.:param spider: the spider which has gone idle:type spider: :class:`~scrapy.spiders.Spider` object+.. note:: Scheduling some requests in your :signal:`spider_idle` handler does+ **not** guarantee that it can prevent the spider from being closed,+ although it sometimes can. That's because the spider may still remain idle+ if all the scheduled requests are rejected by the scheduler (e.g. filtered+ due to duplication).+spider_error------------