Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737)

Andrey Rakhmatullin36bf1185e58e
diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst
index 78b1ab92b..75f15b3ae 100644
--- a/docs/topics/addons.rst
+++ b/docs/topics/addons.rst
@@ -21,10 +21,14 @@ The ``ADDONS`` setting is a dict in which every key is an add-on class or its
import path and the value is its priority.
This is an example where two add-ons are enabled in a project's
-``settings.py``::
+``settings.py``:
+
+.. skip: next
+
+.. code-block:: python
ADDONS = {
- 'path.to.someaddon': 0,
+ "path.to.someaddon": 0,
SomeAddonClass: 1,
}
@@ -56,7 +60,9 @@ the following methods:
:type settings: :class:`~scrapy.settings.BaseSettings`
The settings set by the add-on should use the ``addon`` priority (see
-:ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`)::
+:ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`):
+
+.. code-block:: python
class MyAddon:
def update_settings(self, settings):
diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst
index 05dffcdda..a5ffe00f1 100644
--- a/docs/topics/developer-tools.rst
+++ b/docs/topics/developer-tools.rst
@@ -246,7 +246,6 @@ also request each page to get every quote on the site:
.. code-block:: python
import scrapy
- import json
class QuoteSpider(scrapy.Spider):
@@ -256,7 +255,7 @@ also request each page to get every quote on the site:
start_urls = ["https://quotes.toscrape.com/api/quotes?page=1"]
def parse(self, response):
- data = json.loads(response.text)
+ data = response.json()
for quote in data["quotes"]:
yield {"quote": quote["text"]}
if data["has_next"]:
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index f0b704b9d..ecd154122 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -211,13 +211,17 @@ BaseItemExporter
- ``None`` (all fields [2]_, default)
- - A list of fields::
+ - A list of fields:
- ['field1', 'field2']
+ .. code-block:: python
- - A dict where keys are fields and values are output names::
+ ["field1", "field2"]
- {'field1': 'Field 1', 'field2': 'Field 2'}
+ - A dict where keys are fields and values are output names:
+
+ .. code-block:: python
+
+ {"field1": "Field 1", "field2": "Field 2"}
.. [1] Not all exporters respect the specified field order.
.. [2] When using :ref:`item objects <item-types>` that do not expose
@@ -273,7 +277,9 @@ XmlItemExporter
The additional keyword arguments of this ``__init__`` method are passed to the
:class:`BaseItemExporter` ``__init__`` method.
- A typical output of this exporter would be::
+ A typical output of this exporter would be:
+
+ .. code-block:: xml
<?xml version="1.0" encoding="utf-8"?>
<items>
@@ -291,11 +297,17 @@ XmlItemExporter
exported by serializing each value inside a ``<value>`` element. This is for
convenience, as multi-valued fields are very common.
- For example, the item::
+ For example, the item:
- Item(name=['John', 'Doe'], age='23')
+ .. skip: next
- Would be serialized as::
+ .. code-block:: python
+
+ Item(name=["John", "Doe"], age="23")
+
+ Would be serialized as:
+
+ .. code-block:: xml
<?xml version="1.0" encoding="utf-8"?>
<items>
@@ -379,10 +391,12 @@ PprintItemExporter
The additional keyword arguments of this ``__init__`` method are passed to the
:class:`BaseItemExporter` ``__init__`` method.
- A typical output of this exporter would be::
+ A typical output of this exporter would be:
- {'name': 'Color TV', 'price': '1200'}
- {'name': 'DVD player', 'price': '200'}
+ .. code-block:: python
+
+ {"name": "Color TV", "price": "1200"}
+ {"name": "DVD player", "price": "200"}
Longer lines (when present) are pretty-formatted.
@@ -400,7 +414,9 @@ JsonItemExporter
:param file: the file-like object to use for exporting the data. Its ``write`` method should
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
- A typical output of this exporter would be::
+ A typical output of this exporter would be:
+
+ .. code-block:: json
[{"name": "Color TV", "price": "1200"},
{"name": "DVD player", "price": "200"}]
@@ -429,7 +445,9 @@ JsonLinesItemExporter
:param file: the file-like object to use for exporting the data. Its ``write`` method should
accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
- A typical output of this exporter would be::
+ A typical output of this exporter would be:
+
+ .. code-block:: json
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index 1bab3bfc2..51e142b04 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -434,33 +434,37 @@ This setting is required for enabling the feed export feature.
See :ref:`topics-feed-storage-backends` for supported URI schemes.
-For instance::
+For instance:
+
+.. skip: next
+
+.. code-block:: python
{
- 'items.json': {
- 'format': 'json',
- 'encoding': 'utf8',
- 'store_empty': False,
- 'item_classes': [MyItemClass1, 'myproject.items.MyItemClass2'],
- 'fields': None,
- 'indent': 4,
- 'item_export_kwargs': {
- 'export_empty_fields': True,
+ "items.json": {
+ "format": "json",
+ "encoding": "utf8",
+ "store_empty": False,
+ "item_classes": [MyItemClass1, "myproject.items.MyItemClass2"],
+ "fields": None,
+ "indent": 4,
+ "item_export_kwargs": {
+ "export_empty_fields": True,
},
},
- '/home/user/documents/items.xml': {
- 'format': 'xml',
- 'fields': ['name', 'price'],
- 'item_filter': MyCustomFilter1,
- 'encoding': 'latin1',
- 'indent': 8,
+ "/home/user/documents/items.xml": {
+ "format": "xml",
+ "fields": ["name", "price"],
+ "item_filter": MyCustomFilter1,
+ "encoding": "latin1",
+ "indent": 8,
},
- pathlib.Path('items.csv.gz'): {
- 'format': 'csv',
- 'fields': ['price', 'name'],
- 'item_filter': 'myproject.filters.MyCustomFilter2',
- 'postprocessing': [MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'],
- 'gzip_compresslevel': 5,
+ pathlib.Path("items.csv.gz"): {
+ "format": "csv",
+ "fields": ["price", "name"],
+ "item_filter": "myproject.filters.MyCustomFilter2",
+ "postprocessing": [MyPlugin1, "scrapy.extensions.postprocessing.GzipPlugin"],
+ "gzip_compresslevel": 5,
},
}
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index 3337cc5d0..d4577601d 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -106,10 +106,15 @@ A real example
--------------
Let's see a concrete example of a hypothetical case of memory leaks.
-Suppose we have some spider with a line similar to this one::
+Suppose we have some spider with a line similar to this one:
- return Request(f"http://www.somenastyspider.com/product.php?pid={product_id}",
- callback=self.parse, cb_kwargs={'referer': response})
+.. code-block:: python
+
+ return Request(
+ f"http://www.somenastyspider.com/product.php?pid={product_id}",
+ callback=self.parse,
+ cb_kwargs={"referer": response},
+ )
That line is passing a response reference inside a request which effectively
ties the response lifetime to the requests' one, and that would definitely
diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rst
index 3fc896507..c1d872dd0 100644
--- a/docs/topics/link-extractors.rst
+++ b/docs/topics/link-extractors.rst
@@ -36,7 +36,9 @@ Link extractor reference
The link extractor class is
:class:`scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`. For convenience it
-can also be imported as ``scrapy.linkextractors.LinkExtractor``::
+can also be imported as ``scrapy.linkextractors.LinkExtractor``:
+
+.. code-block:: python
from scrapy.linkextractors import LinkExtractor
diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst
index 765e7f6d7..314e2c1ae 100644
--- a/docs/topics/loaders.rst
+++ b/docs/topics/loaders.rst
@@ -350,7 +350,9 @@ When parsing related values from a subsection of a document, it can be
useful to create nested loaders. Imagine you're extracting details from
a footer of a page that looks something like:
-Example::
+Example:
+
+.. code-block:: html
<footer>
<a class="social" href="https://facebook.com/whatever">Like Us</a>
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index b7a4f561e..56b53d18d 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -470,7 +470,9 @@ When using the Images Pipeline, you can drop images which are too small, by
specifying the minimum allowed size in the :setting:`IMAGES_MIN_HEIGHT` and
:setting:`IMAGES_MIN_WIDTH` settings.
-For example::
+For example:
+
+.. code-block:: python
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
@@ -493,7 +495,9 @@ Allowing redirections
By default media pipelines ignore redirects, i.e. an HTTP redirection
to a media file URL request will mean the media download is considered failed.
-To handle media redirections, set this setting to ``True``::
+To handle media redirections, set this setting to ``True``:
+
+.. code-block:: python
MEDIA_ALLOW_REDIRECTS = True
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index a5f799444..e56be0a22 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -436,7 +436,7 @@ errors if needed:
)
def parse_httpbin(self, response):
- self.logger.info("Got successful response from {}".format(response.url))
+ self.logger.info(f"Got successful response from {response.url}")
# do something useful here...
def errback_httpbin(self, failure):
@@ -1028,9 +1028,13 @@ Response objects
:meth:`~scrapy.http.headers.Headers.get` to return the last header value with
the specified name or :meth:`~scrapy.http.headers.Headers.getlist` to return
all header values with the specified name. For example, this call will give you
- all cookies in the headers::
+ all cookies in the headers:
- response.headers.getlist('Set-Cookie')
+ .. skip: next
+
+ .. code-block:: python
+
+ response.headers.getlist("Set-Cookie")
.. attribute:: Response.body
@@ -1132,7 +1136,11 @@ Response objects
a possible relative url.
This is a wrapper over :func:`~urllib.parse.urljoin`, it's merely an alias for
- making this call::
+ making this call:
+
+ .. skip: next
+
+ .. code-block:: python
urllib.parse.urljoin(response.url, url)
@@ -1221,21 +1229,31 @@ TextResponse objects
.. method:: TextResponse.jmespath(query)
- A shortcut to ``TextResponse.selector.jmespath(query)``::
+ .. skip: start
+
+ A shortcut to ``TextResponse.selector.jmespath(query)``:
- response.jmespath('object.[*]')
+ .. code-block:: python
+
+ response.jmespath("object.[*]")
.. method:: TextResponse.xpath(query)
- A shortcut to ``TextResponse.selector.xpath(query)``::
+ A shortcut to ``TextResponse.selector.xpath(query)``:
+
+ .. code-block:: python
- response.xpath('//p')
+ response.xpath("//p")
.. method:: TextResponse.css(query)
- A shortcut to ``TextResponse.selector.css(query)``::
+ A shortcut to ``TextResponse.selector.css(query)``:
+
+ .. code-block:: python
+
+ response.css("p")
- response.css('p')
+ .. skip: end
.. automethod:: TextResponse.follow
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index a592f8bc5..8068824e3 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -899,7 +899,9 @@ Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting
servers too hard.
Decimal numbers are supported. For example, to send a maximum of 4 requests
-every 10 seconds::
+every 10 seconds:
+
+.. code-block:: python
DOWNLOAD_DELAY = 2.5
@@ -1231,7 +1233,9 @@ the ``dont_filter`` parameter to ``True`` on the ``__init__`` method of a
specific :class:`~scrapy.Request` object that should not be filtered out.
A class assigned to :setting:`DUPEFILTER_CLASS` must implement the following
-interface::
+interface:
+
+.. code-block:: python
class MyDupeFilter:
@@ -1644,21 +1648,6 @@ Default: ``False``
Whether to enable memory debugging.
-.. setting:: MEMDEBUG_NOTIFY
-
-MEMDEBUG_NOTIFY
----------------
-
-Default: ``[]``
-
-When memory debugging is enabled a memory report will be sent to the specified
-addresses if this setting is not empty, otherwise the report will be written to
-the log.
-
-Example::
-
- MEMDEBUG_NOTIFY = ['[email protected]']
-
.. setting:: MEMUSAGE_ENABLED
MEMUSAGE_ENABLED
@@ -1732,9 +1721,11 @@ Default: ``"<project name>.spiders"`` (:ref:`fallback <default-settings>`: ``""`
Module where to create new spiders using the :command:`genspider` command.
-Example::
+Example:
+
+.. code-block:: python
- NEWSPIDER_MODULE = 'mybot.spiders_dev'
+ NEWSPIDER_MODULE = "mybot.spiders_dev"
.. setting:: RANDOMIZE_DOWNLOAD_DELAY
@@ -2209,7 +2200,7 @@ In order to use the reactor installed by Scrapy:
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop("timeout", "60"))
- super(QuotesSpider, self).__init__(*args, **kwargs)
+ super().__init__(*args, **kwargs)
async def start(self):
reactor.callLater(self.timeout, self.stop)
@@ -2238,7 +2229,7 @@ which raises an exception, becomes:
def __init__(self, *args, **kwargs):
self.timeout = int(kwargs.pop("timeout", "60"))
- super(QuotesSpider, self).__init__(*args, **kwargs)
+ super().__init__(*args, **kwargs)
async def start(self):
from twisted.internet import reactor
diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst
index 5b826ed18..ea97a22aa 100644
--- a/docs/topics/shell.rst
+++ b/docs/topics/shell.rst
@@ -34,7 +34,9 @@ is unavailable.
Through Scrapy's settings you can configure it to use any one of
``ipython``, ``bpython`` or the standard ``python`` shell, regardless of which
are installed. This is done by setting the ``SCRAPY_PYTHON_SHELL`` environment
-variable; or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`::
+variable; or by defining it in your :ref:`scrapy.cfg <topics-config-settings>`:
+
+.. code-block:: ini
[settings]
shell = bpython
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 531232149..03996bee6 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -34,7 +34,7 @@ Here is a simple example showing how you can catch signals and perform some acti
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
- spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
+ spider = super().from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
@@ -72,7 +72,7 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
- spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
+ spider = super().from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
return spider
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 770640adf..69ff08fa1 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -308,7 +308,7 @@ Spiders can access arguments in their `__init__` methods:
name = "myspider"
def __init__(self, category=None, *args, **kwargs):
- super(MySpider, self).__init__(*args, **kwargs)
+ super().__init__(*args, **kwargs)
self.start_urls = [f"http://www.example.com/categories/{category}"]
# ...
@@ -622,9 +622,11 @@ XMLFeedSpider
.. attribute:: itertag
- A string with the name of the node (or element) to iterate in. Example::
+ A string with the name of the node (or element) to iterate in. Example:
- itertag = 'product'
+ .. code-block:: python
+
+ itertag = "product"
.. attribute:: namespaces
@@ -637,12 +639,17 @@ XMLFeedSpider
You can then specify nodes with namespaces in the :attr:`itertag`
attribute.
- Example::
+ Example:
+
+ .. code-block:: python
+
+ from scrapy.spiders import XMLFeedSpider
+
class YourSpider(XMLFeedSpider):
- namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
- itertag = 'n:url'
+ namespaces = [("n", "http://www.sitemaps.org/schemas/sitemap/0.9")]
+ itertag = "n:url"
# ...
Apart from these new attributes, this spider has the following overridable
@@ -802,9 +809,11 @@ SitemapSpider
the regular expression. ``callback`` can be a string (indicating the
name of a spider method) or a callable.
- For example::
+ For example:
+
+ .. code-block:: python
- sitemap_rules = [('/product/', 'parse_product')]
+ sitemap_rules = [("/product/", "parse_product")]
Rules are applied in order, and only the first one that matches will be
used.
@@ -826,7 +835,9 @@ SitemapSpider
are links for the same website in another language passed within
the same ``url`` block.
- For example::
+ For example:
+
+ .. code-block:: xml
<url>
<loc>http://example.com/</loc>
@@ -844,7 +855,9 @@ SitemapSpider
This is a filter function that could be overridden to select sitemap entries
based on their attributes.
- For example::
+ For example:
+
+ .. code-block:: xml
<url>
<loc>http://example.com/</loc>
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 3a7dfd018..d283336ab 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -154,7 +154,6 @@ __all__ = [
"MAIL_TLS",
"MAIL_USER",
"MEMDEBUG_ENABLED",
- "MEMDEBUG_NOTIFY",
"MEMUSAGE_CHECK_INTERVAL_SECONDS",
"MEMUSAGE_ENABLED",
"MEMUSAGE_LIMIT_MB",
@@ -470,7 +469,6 @@ MAIL_SSL = False
MAIL_TLS = False
MEMDEBUG_ENABLED = False # enable memory debugging
-MEMDEBUG_NOTIFY = [] # send memory debugging report by mail at engine shutdown
MEMUSAGE_ENABLED = True
MEMUSAGE_CHECK_INTERVAL_SECONDS = 60.0