Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

async/deferred signal handlers (#4390)

Aditya Kumare4750f2fbdac
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 2def53848..8661f86a0 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -16,8 +16,7 @@ deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the
:ref:`topics-api-signals`.
-Here is a simple example showing how you can catch signals and perform some action:
-::
+Here is a simple example showing how you can catch signals and perform some action::
from scrapy import signals
from scrapy import Spider
@@ -52,9 +51,45 @@ Deferred signal handlers
========================
Some signals support returning :class:`~twisted.internet.defer.Deferred`
-objects from their handlers, see the :ref:`topics-signals-ref` below to know
-which ones.
+objects from their handlers, allowing you to run asynchronous code that
+does not block Scrapy. If a signal handler returns a
+:class:`~twisted.internet.defer.Deferred`, Scrapy waits for that
+:class:`~twisted.internet.defer.Deferred` to fire.
+Let's take an example::
+
+ class SignalSpider(scrapy.Spider):
+ name = 'signals'
+ start_urls = ['http://quotes.toscrape.com/page/1/']
+
+ @classmethod
+ def from_crawler(cls, crawler, *args, **kwargs):
+ spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
+ crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
+ return spider
+
+ def item_scraped(self, item):
+ # Send the scraped item to the server
+ d = treq.post(
+ 'http://example.com/post',
+ json.dumps(item).encode('ascii'),
+ headers={b'Content-Type': [b'application/json']}
+ )
+
+ # The next item will be scraped only after
+ # deferred (d) is fired
+ return d
+
+ def parse(self, response):
+ for quote in response.css('div.quote'):
+ yield {
+ 'text': quote.css('span.text::text').get(),
+ 'author': quote.css('small.author::text').get(),
+ 'tags': quote.css('div.tags a.tag::text').getall(),
+ }
+
+See the :ref:`topics-signals-ref` below to know which signals support
+:class:`~twisted.internet.defer.Deferred`.
.. _topics-signals-ref:
@@ -66,9 +101,12 @@ Built-in signals reference
Here's the list of Scrapy built-in signals and their meaning.
-engine_started
+Engine signals
--------------
+engine_started
+~~~~~~~~~~~~~~
+
.. signal:: engine_started
.. function:: engine_started()
@@ -81,7 +119,7 @@ engine_started
getting fired before :signal:`spider_opened`.
engine_stopped
---------------
+~~~~~~~~~~~~~~
.. signal:: engine_stopped
.. function:: engine_stopped()
@@ -91,9 +129,20 @@ engine_stopped
This signal supports returning deferreds from their handlers.
-item_scraped
+Item signals
------------
+.. note::
+ As at max :setting:`CONCURRENT_ITEMS` items are processed in
+ parallel, many deferreds are fired together using
+ :class:`~twisted.internet.defer.DeferredList`. Hence the next
+ batch waits for the :class:`~twisted.internet.defer.DeferredList`
+ to fire and then runs the respective item signal handler for
+ the next batch of scraped items.
+
+item_scraped
+~~~~~~~~~~~~
+
.. signal:: item_scraped
.. function:: item_scraped(item, response, spider)
@@ -112,7 +161,7 @@ item_scraped
:type response: :class:`~scrapy.http.Response` object
item_dropped
-------------
+~~~~~~~~~~~~
.. signal:: item_dropped
.. function:: item_dropped(item, response, exception, spider)
@@ -137,7 +186,7 @@ item_dropped
:type exception: :exc:`~scrapy.exceptions.DropItem` exception
item_error
-------------
+~~~~~~~~~~
.. signal:: item_error
.. function:: item_error(item, response, spider, failure)
@@ -159,8 +208,11 @@ item_error
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
+Spider signals
+--------------
+
spider_closed
--------------
+~~~~~~~~~~~~~
.. signal:: spider_closed
.. function:: spider_closed(spider, reason)
@@ -183,7 +235,7 @@ spider_closed
:type reason: str
spider_opened
--------------
+~~~~~~~~~~~~~
.. signal:: spider_opened
.. function:: spider_opened(spider)
@@ -198,7 +250,7 @@ spider_opened
:type spider: :class:`~scrapy.spiders.Spider` object
spider_idle
------------
+~~~~~~~~~~~
.. signal:: spider_idle
.. function:: spider_idle(spider)
@@ -228,7 +280,7 @@ spider_idle
due to duplication).
spider_error
-------------
+~~~~~~~~~~~~
.. signal:: spider_error
.. function:: spider_error(failure, response, spider)
@@ -246,8 +298,11 @@ spider_error
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.spiders.Spider` object
+Request signals
+---------------
+
request_scheduled
------------------
+~~~~~~~~~~~~~~~~~
.. signal:: request_scheduled
.. function:: request_scheduled(request, spider)
@@ -264,7 +319,7 @@ request_scheduled
:type spider: :class:`~scrapy.spiders.Spider` object
request_dropped
----------------
+~~~~~~~~~~~~~~~
.. signal:: request_dropped
.. function:: request_dropped(request, spider)
@@ -281,7 +336,7 @@ request_dropped
:type spider: :class:`~scrapy.spiders.Spider` object
request_reached_downloader
----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
.. signal:: request_reached_downloader
.. function:: request_reached_downloader(request, spider)
@@ -297,7 +352,7 @@ request_reached_downloader
:type spider: :class:`~scrapy.spiders.Spider` object
request_left_downloader
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
.. signal:: request_left_downloader
.. function:: request_left_downloader(request, spider)
@@ -315,8 +370,11 @@ request_left_downloader
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
+Response signals
+----------------
+
response_received
------------------
+~~~~~~~~~~~~~~~~~
.. signal:: response_received
.. function:: response_received(response, request, spider)
@@ -336,7 +394,7 @@ response_received
:type spider: :class:`~scrapy.spiders.Spider` object
response_downloaded
--------------------
+~~~~~~~~~~~~~~~~~~~
.. signal:: response_downloaded
.. function:: response_downloaded(response, request, spider)