Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge branch 'master' into bytes_received_signal
Eugenio Lacuesta4d625afda5d1
diff --cc docs/topics/signals.rstindex 02fa6e287,8661f86a0..bc04faad5--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@@ -89,11 -127,22 +127,22 @@@ engine_stoppeSent when the Scrapy engine is stopped (for example, when a crawlingprocess has finished).- This signal supports returning deferreds from their handlers.+ This signal supports returning deferreds from its handlers.- item_scraped+ Item signals------------+ .. note::+ As at max :setting:`CONCURRENT_ITEMS` items are processed in+ parallel, many deferreds are fired together using+ :class:`~twisted.internet.defer.DeferredList`. Hence the next+ batch waits for the :class:`~twisted.internet.defer.DeferredList`+ to fire and then runs the respective item signal handler for+ the next batch of scraped items.++ item_scraped+ ~~~~~~~~~~~~+.. signal:: item_scraped.. function:: item_scraped(item, response, spider)@@@ -336,8 -370,11 +370,32 @@@ request_left_downloade:param spider: the spider that yielded the request:type spider: :class:`~scrapy.spiders.Spider` object++bytes_received++~~~~~~~~~~~~~~++++.. signal:: bytes_received++.. function:: bytes_received(data, request, spider)++++ Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is++ received for a specific request. This signal might be fired multiple++ times for the same request, with partial data each time.++++ This signal does not support returning deferreds from its handlers.++++ :param data: the data received by the download handler++ :type spider: :class:`bytes` object++++ :param request: the request that generated the response++ :type request: :class:`~scrapy.http.Request` object++++ :param spider: the spider associated with the response++ :type spider: :class:`~scrapy.spiders.Spider` object+++ Response signals+ ----------------+response_received- -----------------+ ~~~~~~~~~~~~~~~~~.. signal:: response_received.. function:: response_received(response, request, spider)diff --cc scrapy/core/downloader/handlers/http11.pyindex bda21b6b9,9be8ffdfb..c21491f52--- a/scrapy/core/downloader/handlers/http11.py+++ b/scrapy/core/downloader/handlers/http11.py@@@ -476,7 -461,7 +474,8 @@@ class _ResponseReader(protocol.Protocolself._reached_warnsize = Falseself._bytes_received = 0self._certificate = None+ self._ip_address = None+ self._crawler = crawlerdef connectionMade(self):if self._certificate is None: