Scrapy, a fast high-level web crawling & scraping framework for Python.
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737)
Andrey Rakhmatullin36bf1185e58e
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rstindex 78b1ab92b..75f15b3ae 100644--- a/docs/topics/addons.rst+++ b/docs/topics/addons.rst@@ -21,10 +21,14 @@ The ``ADDONS`` setting is a dict in which every key is an add-on class or itsimport path and the value is its priority.This is an example where two add-ons are enabled in a project's-``settings.py``::+``settings.py``:++.. skip: next++.. code-block:: pythonADDONS = {- 'path.to.someaddon': 0,+ "path.to.someaddon": 0,SomeAddonClass: 1,}@@ -56,7 +60,9 @@ the following methods::type settings: :class:`~scrapy.settings.BaseSettings`The settings set by the add-on should use the ``addon`` priority (see-:ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`)::+:ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`):++.. code-block:: pythonclass MyAddon:def update_settings(self, settings):diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex 05dffcdda..a5ffe00f1 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -246,7 +246,6 @@ also request each page to get every quote on the site:.. code-block:: pythonimport scrapy- import jsonclass QuoteSpider(scrapy.Spider):@@ -256,7 +255,7 @@ also request each page to get every quote on the site:start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]def parse(self, response):- data = json.loads(response.text)+ data = response.json()for quote in data["quotes"]:yield {"quote": quote["text"]}if data["has_next"]:diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex f0b704b9d..ecd154122 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -211,13 +211,17 @@ BaseItemExporter- ``None`` (all fields [2]_, default)- - A list of fields::+ - A list of fields:- ['field1', 'field2']+ .. code-block:: python- - A dict where keys are fields and values are output names::+ ["field1", "field2"]- {'field1': 'Field 1', 'field2': 'Field 2'}+ - A dict where keys are fields and values are output names:++ .. code-block:: python++ {"field1": "Field 1", "field2": "Field 2"}.. [1] Not all exporters respect the specified field order... [2] When using :ref:`item objects <item-types>` that do not expose@@ -273,7 +277,9 @@ XmlItemExporterThe additional keyword arguments of this ``__init__`` method are passed to the:class:`BaseItemExporter` ``__init__`` method.- A typical output of this exporter would be::+ A typical output of this exporter would be:++ .. code-block:: xml<?xml version="1.0" encoding="utf-8"?><items>@@ -291,11 +297,17 @@ XmlItemExporterexported by serializing each value inside a ``<value>`` element. This is forconvenience, as multi-valued fields are very common.- For example, the item::+ For example, the item:- Item(name=['John', 'Doe'], age='23')+ .. skip: next- Would be serialized as::+ .. code-block:: python++ Item(name=["John", "Doe"], age="23")++ Would be serialized as:++ .. code-block:: xml<?xml version="1.0" encoding="utf-8"?><items>@@ -379,10 +391,12 @@ PprintItemExporterThe additional keyword arguments of this ``__init__`` method are passed to the:class:`BaseItemExporter` ``__init__`` method.- A typical output of this exporter would be::+ A typical output of this exporter would be:- {'name': 'Color TV', 'price': '1200'}- {'name': 'DVD player', 'price': '200'}+ .. code-block:: python++ {"name": "Color TV", "price": "1200"}+ {"name": "DVD player", "price": "200"}Longer lines (when present) are pretty-formatted.@@ -400,7 +414,9 @@ JsonItemExporter:param file: the file-like object to use for exporting the data. Its ``write`` method shouldaccept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)- A typical output of this exporter would be::+ A typical output of this exporter would be:++ .. code-block:: json[{"name": "Color TV", "price": "1200"},{"name": "DVD player", "price": "200"}]@@ -429,7 +445,9 @@ JsonLinesItemExporter:param file: the file-like object to use for exporting the data. Its ``write`` method shouldaccept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)- A typical output of this exporter would be::+ A typical output of this exporter would be:++ .. code-block:: json{"name": "Color TV", "price": "1200"}{"name": "DVD player", "price": "200"}diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 1bab3bfc2..51e142b04 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -434,33 +434,37 @@ This setting is required for enabling the feed export feature.See :ref:`topics-feed-storage-backends` for supported URI schemes.-For instance::+For instance:++.. skip: next++.. code-block:: python{- 'items.json': {- 'format': 'json',- 'encoding': 'utf8',- 'store_empty': False,- 'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'],- 'fields': None,- 'indent': 4,- 'item_export_kwargs': {- 'export_empty_fields': True,+ "items.json": {+ "format": "json",+ "encoding": "utf8",+ "store_empty": False,+ "item_classes": [MyItemClass1, "myproject.items.MyItemClass2"],+ "fields": None,+ "indent": 4,+ "item_export_kwargs": {+ "export_empty_fields": True,},},- '/home/user/documents/items.xml': {- 'format': 'xml',- 'fields': ['name', 'price'],- 'item_filter': MyCustomFilter1,- 'encoding': 'latin1',- 'indent': 8,+ "/home/user/documents/items.xml": {+ "format": "xml",+ "fields": ["name", "price"],+ "item_filter": MyCustomFilter1,+ "encoding": "latin1",+ "indent": 8,},- pathlib.Path('items.csv.gz'): {- 'format': 'csv',- 'fields': ['price', 'name'],- 'item_filter': 'myproject.filters.MyCustomFilter2',- 'postprocessing': [MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'],- 'gzip_compresslevel': 5,+ pathlib.Path("items.csv.gz"): {+ "format": "csv",+ "fields": ["price", "name"],+ "item_filter": "myproject.filters.MyCustomFilter2",+ "postprocessing": [MyPlugin1, "scrapy.extensions.postprocessing.GzipPlugin"],+ "gzip_compresslevel": 5,},}diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rstindex 3337cc5d0..d4577601d 100644--- a/docs/topics/leaks.rst+++ b/docs/topics/leaks.rst@@ -106,10 +106,15 @@ A real example--------------Let's see a concrete example of a hypothetical case of memory leaks.-Suppose we have some spider with a line similar to this one::+Suppose we have some spider with a line similar to this one:- return Request(f"http://www.somenastyspider.com/product.php?pid={product_id}",- callback=self.parse, cb_kwargs={'referer': response})+.. code-block:: python++ return Request(+ f"http://www.somenastyspider.com/product.php?pid={product_id}",+ callback=self.parse,+ cb_kwargs={"referer": response},+ )That line is passing a response reference inside a request which effectivelyties the response lifetime to the requests' one, and that would definitelydiff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rstindex 3fc896507..c1d872dd0 100644--- a/docs/topics/link-extractors.rst+++ b/docs/topics/link-extractors.rst@@ -36,7 +36,9 @@ Link extractor referenceThe link extractor class is:class:`scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`. For convenience it-can also be imported as ``scrapy.linkextractors.LinkExtractor``::+can also be imported as ``scrapy.linkextractors.LinkExtractor``:++.. code-block:: pythonfrom scrapy.linkextractors import LinkExtractordiff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rstindex 765e7f6d7..314e2c1ae 100644--- a/docs/topics/loaders.rst+++ b/docs/topics/loaders.rst@@ -350,7 +350,9 @@ When parsing related values from a subsection of a document, it can beuseful to create nested loaders. Imagine you're extracting details froma footer of a page that looks something like:-Example::+Example:++.. code-block:: html<footer><a class="social" href="https://facebook.com/whatever">Like Us</a>diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex b7a4f561e..56b53d18d 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -470,7 +470,9 @@ When using the Images Pipeline, you can drop images which are too small, byspecifying the minimum allowed size in the :setting:`IMAGES_MIN_HEIGHT` and:setting:`IMAGES_MIN_WIDTH` settings.-For example::+For example:++.. code-block:: pythonIMAGES_MIN_HEIGHT = 110IMAGES_MIN_WIDTH = 110@@ -493,7 +495,9 @@ Allowing redirectionsBy default media pipelines ignore redirects, i.e. an HTTP redirectionto a media file URL request will mean the media download is considered failed.-To handle media redirections, set this setting to ``True``::+To handle media redirections, set this setting to ``True``:++.. code-block:: pythonMEDIA_ALLOW_REDIRECTS = Truediff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex a5f799444..e56be0a22 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -436,7 +436,7 @@ errors if needed:)def parse_httpbin(self, response):- self.logger.info("Got successful response from {}".format(response.url))+ self.logger.info(f"Got successful response from {response.url}")# do something useful here...def errback_httpbin(self, failure):@@ -1028,9 +1028,13 @@ Response objects:meth:`~scrapy.http.headers.Headers.get` to return the last header value withthe specified name or :meth:`~scrapy.http.headers.Headers.getlist` to returnall header values with the specified name. For example, this call will give you- all cookies in the headers::+ all cookies in the headers:- response.headers.getlist('Set-Cookie')+ .. skip: next++ .. code-block:: python++ response.headers.getlist("Set-Cookie").. attribute:: Response.body@@ -1132,7 +1136,11 @@ Response objectsa possible relative url.This is a wrapper over :func:`~urllib.parse.urljoin`, it's merely an alias for- making this call::+ making this call:++ .. skip: next++ .. code-block:: pythonurllib.parse.urljoin(response.url, url)@@ -1221,21 +1229,31 @@ TextResponse objects.. method:: TextResponse.jmespath(query)- A shortcut to ``TextResponse.selector.jmespath(query)``::+ .. skip: start++ A shortcut to ``TextResponse.selector.jmespath(query)``:- response.jmespath('object.[*]')+ .. code-block:: python++ response.jmespath("object.[*]").. method:: TextResponse.xpath(query)- A shortcut to ``TextResponse.selector.xpath(query)``::+ A shortcut to ``TextResponse.selector.xpath(query)``:++ .. code-block:: python- response.xpath('//p')+ response.xpath("//p").. method:: TextResponse.css(query)- A shortcut to ``TextResponse.selector.css(query)``::+ A shortcut to ``TextResponse.selector.css(query)``:++ .. code-block:: python++ response.css("p")- response.css('p')+ .. skip: end.. automethod:: TextResponse.followdiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex a592f8bc5..8068824e3 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -899,7 +899,9 @@ Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hittingservers too hard.Decimal numbers are supported. For example, to send a maximum of 4 requests-every 10 seconds::+every 10 seconds:++.. code-block:: pythonDOWNLOAD_DELAY = 2.5@@ -1231,7 +1233,9 @@ the ``dont_filter`` parameter to ``True`` on the ``__init__`` method of aspecific :class:`~scrapy.Request` object that should not be filtered out.A class assigned to :setting:`DUPEFILTER_CLASS` must implement the following-interface::+interface:++.. code-block:: pythonclass MyDupeFilter:@@ -1644,21 +1648,6 @@ Default: ``False``Whether to enable memory debugging.-.. setting:: MEMDEBUG_NOTIFY--MEMDEBUG_NOTIFY------------------Default: ``[]``--When memory debugging is enabled a memory report will be sent to the specified-addresses if this setting is not empty, otherwise the report will be written to-the log.--Example::-- MEMDEBUG_NOTIFY = ['[email protected]']-.. setting:: MEMUSAGE_ENABLEDMEMUSAGE_ENABLED@@ -1732,9 +1721,11 @@ Default: ``"<project name>.spiders"`` (:ref:`fallback <default-settings>`: ``""`Module where to create new spiders using the :command:`genspider` command.-Example::+Example:++.. code-block:: python- NEWSPIDER_MODULE = 'mybot.spiders_dev'+ NEWSPIDER_MODULE = "mybot.spiders_dev".. setting:: RANDOMIZE_DOWNLOAD_DELAY@@ -2209,7 +2200,7 @@ In order to use the reactor installed by Scrapy:def __init__(self, *args, **kwargs):self.timeout = int(kwargs.pop("timeout", "60"))- super(QuotesSpider, self).__init__(*args, **kwargs)+ super().__init__(*args, **kwargs)async def start(self):reactor.callLater(self.timeout, self.stop)@@ -2238,7 +2229,7 @@ which raises an exception, becomes:def __init__(self, *args, **kwargs):self.timeout = int(kwargs.pop("timeout", "60"))- super(QuotesSpider, self).__init__(*args, **kwargs)+ super().__init__(*args, **kwargs)async def start(self):from twisted.internet import reactordiff --git a/docs/topics/shell.rst b/docs/topics/shell.rstindex 5b826ed18..ea97a22aa 100644--- a/docs/topics/shell.rst+++ b/docs/topics/shell.rst@@ -34,7 +34,9 @@ is unavailable.Through Scrapy's settings you can configure it to use any one of``ipython``, ``bpython`` or the standard ``python`` shell, regardless of whichare installed. This is done by setting the ``SCRAPY_PYTHON_SHELL`` environment-variable; or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`::+variable; or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`:++.. code-block:: ini[settings]shell = bpythondiff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 531232149..03996bee6 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -34,7 +34,7 @@ Here is a simple example showing how you can catch signals and perform some acti@classmethoddef from_crawler(cls, crawler, *args, **kwargs):- spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)+ spider = super().from_crawler(crawler, *args, **kwargs)crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)return spider@@ -72,7 +72,7 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:@classmethoddef from_crawler(cls, crawler, *args, **kwargs):- spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)+ spider = super().from_crawler(crawler, *args, **kwargs)crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)return spiderdiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 770640adf..69ff08fa1 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -308,7 +308,7 @@ Spiders can access arguments in their `__init__` methods:name = "myspider"def __init__(self, category=None, *args, **kwargs):- super(MySpider, self).__init__(*args, **kwargs)+ super().__init__(*args, **kwargs)self.start_urls = [f"http://www.example.com/categories/{category}"]# ...@@ -622,9 +622,11 @@ XMLFeedSpider.. attribute:: itertag- A string with the name of the node (or element) to iterate in. Example::+ A string with the name of the node (or element) to iterate in. Example:- itertag = 'product'+ .. code-block:: python++ itertag = "product".. attribute:: namespaces@@ -637,12 +639,17 @@ XMLFeedSpiderYou can then specify nodes with namespaces in the :attr:`itertag`attribute.- Example::+ Example:++ .. code-block:: python++ from scrapy.spiders import XMLFeedSpider+class YourSpider(XMLFeedSpider):- namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]- itertag = 'n:url'+ namespaces = [("n", "http://www.sitemaps.org/schemas/sitemap/0.9")]+ itertag = "n:url"# ...Apart from these new attributes, this spider has the following overridable@@ -802,9 +809,11 @@ SitemapSpiderthe regular expression. ``callback`` can be a string (indicating thename of a spider method) or a callable.- For example::+ For example:++ .. code-block:: python- sitemap_rules = [('/product/', 'parse_product')]+ sitemap_rules = [("/product/", "parse_product")]Rules are applied in order, and only the first one that matches will beused.@@ -826,7 +835,9 @@ SitemapSpiderare links for the same website in another language passed withinthe same ``url`` block.- For example::+ For example:++ .. code-block:: xml<url><loc>http://example.com/</loc>@@ -844,7 +855,9 @@ SitemapSpiderThis is a filter function that could be overridden to select sitemap entriesbased on their attributes.- For example::+ For example:++ .. code-block:: xml<url><loc>http://example.com/</loc>diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.pyindex 3a7dfd018..d283336ab 100644--- a/scrapy/settings/default_settings.py+++ b/scrapy/settings/default_settings.py@@ -154,7 +154,6 @@ __all__ = ["MAIL_TLS","MAIL_USER","MEMDEBUG_ENABLED",- "MEMDEBUG_NOTIFY","MEMUSAGE_CHECK_INTERVAL_SECONDS","MEMUSAGE_ENABLED","MEMUSAGE_LIMIT_MB",@@ -470,7 +469,6 @@ MAIL_SSL = FalseMAIL_TLS = FalseMEMDEBUG_ENABLED = False # enable memory debugging-MEMDEBUG_NOTIFY = [] # send memory debugging report by mail at engine shutdownMEMUSAGE_ENABLED = TrueMEMUSAGE_CHECK_INTERVAL_SECONDS = 60.0