Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge branch 'master' into bytes_received_signal

Eugenio Lacuesta4d625afda5d1
diff --cc docs/topics/signals.rst
index 02fa6e287,8661f86a0..bc04faad5
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@@ -89,11 -127,22 +127,22 @@@ engine_stoppe
Sent when the Scrapy engine is stopped (for example, when a crawling
process has finished).
- This signal supports returning deferreds from their handlers.
+ This signal supports returning deferreds from its handlers.
- item_scraped
+ Item signals
------------
+ .. note::
+ As at max :setting:`CONCURRENT_ITEMS` items are processed in
+ parallel, many deferreds are fired together using
+ :class:`~twisted.internet.defer.DeferredList`. Hence the next
+ batch waits for the :class:`~twisted.internet.defer.DeferredList`
+ to fire and then runs the respective item signal handler for
+ the next batch of scraped items.
+
+ item_scraped
+ ~~~~~~~~~~~~
+
.. signal:: item_scraped
.. function:: item_scraped(item, response, spider)
@@@ -336,8 -370,11 +370,32 @@@ request_left_downloade
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.spiders.Spider` object
++bytes_received
++~~~~~~~~~~~~~~
++
++.. signal:: bytes_received
++.. function:: bytes_received(data, request, spider)
++
++ Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
++ received for a specific request. This signal might be fired multiple
++ times for the same request, with partial data each time.
++
++ This signal does not support returning deferreds from its handlers.
++
++ :param data: the data received by the download handler
++ :type spider: :class:`bytes` object
++
++ :param request: the request that generated the response
++ :type request: :class:`~scrapy.http.Request` object
++
++ :param spider: the spider associated with the response
++ :type spider: :class:`~scrapy.spiders.Spider` object
++
+ Response signals
+ ----------------
+
response_received
- -----------------
+ ~~~~~~~~~~~~~~~~~
.. signal:: response_received
.. function:: response_received(response, request, spider)
diff --cc scrapy/core/downloader/handlers/http11.py
index bda21b6b9,9be8ffdfb..c21491f52
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@@ -476,7 -461,7 +474,8 @@@ class _ResponseReader(protocol.Protocol
self._reached_warnsize = False
self._bytes_received = 0
self._certificate = None
+ self._ip_address = None
+ self._crawler = crawler
def connectionMade(self):
if self._certificate is None: