Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge branch 'master' into bytes_received_signal

Eugenio Lacuesta21f1eddc90fa
diff --cc docs/topics/signals.rst
index dfb87cef3,2def53848..97be46f2a
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@@ -141,10 -142,10 +142,10 @@@ item_erro
.. signal:: item_error
.. function:: item_error(item, response, spider, failure)
- Sent when a :ref:`topics-item-pipeline` generates an error (ie. raises
+ Sent when a :ref:`topics-item-pipeline` generates an error (i.e. raises
an exception), except :exc:`~scrapy.exceptions.DropItem` exception.
- This signal supports returning deferreds from their handlers.
+ This signal supports returning deferreds from its handlers.
:param item: the item dropped from the :ref:`topics-item-pipeline`
:type item: dict or :class:`~scrapy.item.Item` object
@@@ -257,9 -233,9 +258,9 @@@ spider_erro
.. signal:: spider_error
.. function:: spider_error(failure, response, spider)
- Sent when a spider callback generates an error (ie. raises an exception).
+ Sent when a spider callback generates an error (i.e. raises an exception).
- This signal does not support returning deferreds from their handlers.
+ This signal does not support returning deferreds from its handlers.
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
diff --cc scrapy/core/downloader/handlers/http11.py
index 7a1a77b23,93951d3b5..93fdb20a8
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@@ -474,9 -458,13 +476,15 @@@ class _ResponseReader(protocol.Protocol
self._fail_on_dataloss_warned = False
self._reached_warnsize = False
self._bytes_received = 0
+ self._certificate = None
+ self._crawler = crawler
+ self._source = source
+ def connectionMade(self):
+ if self._certificate is None:
+ with suppress(AttributeError):
+ self._certificate = ssl.Certificate(self.transport._producer.getPeerCertificate())
+
def dataReceived(self, bodyBytes):
# This maybe called several times after cancel was called with buffered data.
if self._finished.called:
diff --cc scrapy/signals.py
index 590421893,cd7ed7fb1..c61ae6ec3
--- a/scrapy/signals.py
+++ b/scrapy/signals.py
@@@ -14,9 -14,9 +14,10 @@@ spider_error = object(
request_scheduled = object()
request_dropped = object()
request_reached_downloader = object()
+ request_left_downloader = object()
response_received = object()
response_downloaded = object()
+bytes_received = object()
item_scraped = object()
item_dropped = object()
item_error = object()