Scrapy, a fast high-level web crawling & scraping framework for Python.
async/deferred signal handlers (#4390)
Aditya Kumare4750f2fbdac
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 2def53848..8661f86a0 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -16,8 +16,7 @@ deliver the arguments that the handler receives.You can connect to signals (or send your own) through the:ref:`topics-api-signals`.-Here is a simple example showing how you can catch signals and perform some action:-::+Here is a simple example showing how you can catch signals and perform some action::from scrapy import signalsfrom scrapy import Spider@@ -52,9 +51,45 @@ Deferred signal handlers========================Some signals support returning :class:`~twisted.internet.defer.Deferred`-objects from their handlers, see the :ref:`topics-signals-ref` below to know-which ones.+objects from their handlers, allowing you to run asynchronous code that+does not block Scrapy. If a signal handler returns a+:class:`~twisted.internet.defer.Deferred`, Scrapy waits for that+:class:`~twisted.internet.defer.Deferred` to fire.+Let's take an example::++ class SignalSpider(scrapy.Spider):+ name = 'signals'+ start_urls = ['http://quotes.toscrape.com/page/1/']++ @classmethod+ def from_crawler(cls, crawler, *args, **kwargs):+ spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)+ crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)+ return spider++ def item_scraped(self, item):+ # Send the scraped item to the server+ d = treq.post(+ 'http://example.com/post',+ json.dumps(item).encode('ascii'),+ headers={b'Content-Type': [b'application/json']}+ )++ # The next item will be scraped only after+ # deferred (d) is fired+ return d++ def parse(self, response):+ for quote in response.css('div.quote'):+ yield {+ 'text': quote.css('span.text::text').get(),+ 'author': quote.css('small.author::text').get(),+ 'tags': quote.css('div.tags a.tag::text').getall(),+ }++See the :ref:`topics-signals-ref` below to know which signals support+:class:`~twisted.internet.defer.Deferred`... _topics-signals-ref:@@ -66,9 +101,12 @@ Built-in signals referenceHere's the list of Scrapy built-in signals and their meaning.-engine_started+Engine signals--------------+engine_started+~~~~~~~~~~~~~~+.. signal:: engine_started.. function:: engine_started()@@ -81,7 +119,7 @@ engine_startedgetting fired before :signal:`spider_opened`.engine_stopped---------------+~~~~~~~~~~~~~~.. signal:: engine_stopped.. function:: engine_stopped()@@ -91,9 +129,20 @@ engine_stoppedThis signal supports returning deferreds from their handlers.-item_scraped+Item signals------------+.. note::+ As at max :setting:`CONCURRENT_ITEMS` items are processed in+ parallel, many deferreds are fired together using+ :class:`~twisted.internet.defer.DeferredList`. Hence the next+ batch waits for the :class:`~twisted.internet.defer.DeferredList`+ to fire and then runs the respective item signal handler for+ the next batch of scraped items.++item_scraped+~~~~~~~~~~~~+.. signal:: item_scraped.. function:: item_scraped(item, response, spider)@@ -112,7 +161,7 @@ item_scraped:type response: :class:`~scrapy.http.Response` objectitem_dropped-------------+~~~~~~~~~~~~.. signal:: item_dropped.. function:: item_dropped(item, response, exception, spider)@@ -137,7 +186,7 @@ item_dropped:type exception: :exc:`~scrapy.exceptions.DropItem` exceptionitem_error-------------+~~~~~~~~~~.. signal:: item_error.. function:: item_error(item, response, spider, failure)@@ -159,8 +208,11 @@ item_error:param failure: the exception raised:type failure: twisted.python.failure.Failure+Spider signals+--------------+spider_closed--------------+~~~~~~~~~~~~~.. signal:: spider_closed.. function:: spider_closed(spider, reason)@@ -183,7 +235,7 @@ spider_closed:type reason: strspider_opened--------------+~~~~~~~~~~~~~.. signal:: spider_opened.. function:: spider_opened(spider)@@ -198,7 +250,7 @@ spider_opened:type spider: :class:`~scrapy.spiders.Spider` objectspider_idle------------+~~~~~~~~~~~.. signal:: spider_idle.. function:: spider_idle(spider)@@ -228,7 +280,7 @@ spider_idledue to duplication).spider_error-------------+~~~~~~~~~~~~.. signal:: spider_error.. function:: spider_error(failure, response, spider)@@ -246,8 +298,11 @@ spider_error:param spider: the spider which raised the exception:type spider: :class:`~scrapy.spiders.Spider` object+Request signals+---------------+request_scheduled------------------+~~~~~~~~~~~~~~~~~.. signal:: request_scheduled.. function:: request_scheduled(request, spider)@@ -264,7 +319,7 @@ request_scheduled:type spider: :class:`~scrapy.spiders.Spider` objectrequest_dropped----------------+~~~~~~~~~~~~~~~.. signal:: request_dropped.. function:: request_dropped(request, spider)@@ -281,7 +336,7 @@ request_dropped:type spider: :class:`~scrapy.spiders.Spider` objectrequest_reached_downloader----------------------------+~~~~~~~~~~~~~~~~~~~~~~~~~~.. signal:: request_reached_downloader.. function:: request_reached_downloader(request, spider)@@ -297,7 +352,7 @@ request_reached_downloader:type spider: :class:`~scrapy.spiders.Spider` objectrequest_left_downloader------------------------+~~~~~~~~~~~~~~~~~~~~~~~.. signal:: request_left_downloader.. function:: request_left_downloader(request, spider)@@ -315,8 +370,11 @@ request_left_downloader:param spider: the spider that yielded the request:type spider: :class:`~scrapy.spiders.Spider` object+Response signals+----------------+response_received------------------+~~~~~~~~~~~~~~~~~.. signal:: response_received.. function:: response_received(response, request, spider)@@ -336,7 +394,7 @@ response_received:type spider: :class:`~scrapy.spiders.Spider` objectresponse_downloaded--------------------+~~~~~~~~~~~~~~~~~~~.. signal:: response_downloaded.. function:: response_downloaded(response, request, spider)