Scrapy, a fast high-level web crawling & scraping framework for Python.
Add release notes for Scrapy 2.20.0 (#8462)
Adrianb1849344da77
diff --git a/docs/news.rst b/docs/news.rstindex 07df72a52..a6dac942c 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,8 +3,17 @@Release notes=============-Scrapy VERSION (unreleased)----------------------------+Scrapy 2.20.0 (unreleased)+--------------------------++Highlights:++- Web-browser-like detection of response encodings, base URLs and meta+ refresh redirects, powered by frostwork_++- Requests can be awaited to get their response++- Non-asyncio Twisted reactors are deprecatedSecurity bug fixes~~~~~~~~~~~~~~~~~~@@ -12,46 +21,124 @@ Security bug fixes- Scrapy now limits by default the memory that responses can take, fixing`CVE-2017-14158`_.- In previous Scrapy versions, that memory had no limit. For example, when- :ref:`media pipelines <topics-media-pipeline>` downloaded files faster than- they could store them, e.g. to Amazon S3, the downloaded files piled up in- memory, so a crawled website could make Scrapy run out of memory.-- To change or disable the limit, use the new- :setting:`RESPONSE_MAX_ACTIVE_SIZE` setting.+ In previous Scrapy versions, the only limit on the total size of responses+ in memory, ``SCRAPER_SLOT_MAX_ACTIVE_SIZE``, covered responses waiting for+ or going through spider callbacks. Responses to other requests, like those+ of :ref:`media pipelines <topics-media-pipeline>`, had no limit. For+ example, when media pipelines downloaded files faster than they could store+ them, e.g. to Amazon S3, the downloaded files piled up in memory, so a+ crawled website could make Scrapy run out of memory.++ The new :setting:`RESPONSE_MAX_ACTIVE_SIZE` setting replaces+ ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` and covers all responses in memory,+ including those to :ref:`awaited requests <inline-requests>` and to+ requests sent with :meth:`ExecutionEngine.download_async()+ <scrapy.core.engine.ExecutionEngine.download_async>`. Use it to change or+ disable the limit. The :command:`shell` disables it... _CVE-2017-14158: https://github.com/advisories/GHSA-h7wm-ph43-c39p+ (:gh:`482`, :gh:`4583`, :gh:`6430`, :gh:`8417`, :gh:`8459`, :gh:`8460`)++Modified requirements+~~~~~~~~~~~~~~~~~~~~~++- Dropped support for Python 3.10.+ (:gh:`8261`)++- Dropped support for PyPy.+ (:gh:`7849`, :gh:`8358`, :gh:`8367`, :gh:`8393`)++- New dependencies:++ - aiodns_ >= 3.3.0++ - aioftp_ >= 0.26.1++ - formerly_ >= 0.1.0++ - frostwork_ >= 0.2.0++ - typing-extensions_ >= 4.6.0 (on Python 3.12 and lower)++ (:gh:`7894`, :gh:`8213`, :gh:`8303`, :gh:`8376`, :gh:`8378`, :gh:`8393`)++- charset-normalizer_ is no longer a dependency.+ (:gh:`8393`)++- Increased the minimum versions of the following dependencies:++ - cssselect_: 0.9.1 → 1.2.0++ - itemloaders_: 1.0.1 → 1.0.5++ - lxml_: 4.6.4 → 4.9.4++ - parsel_: 1.5.0 → 1.8.1++ - Protego_: 0.1.15 → 0.2.0++ - queuelib_: 1.6.1 → 1.8.0++ - Twisted_: 21.7.0 → 23.10.0++ - w3lib_: 2.1.1 → 2.6.0++ (:gh:`6430`, :gh:`8195`, :gh:`8261`, :gh:`8303`, :gh:`8393`, :gh:`8460`)++- Changes to :ref:`extras <extras>`:++ - New ``clipboard``, ``memusage`` and ``socks`` extras.++ - ``gcs``: google-cloud-storage_ 1.29.0 → 1.30.0++ - ``httpx``: httpx2_ 2.0.0 → 2.3.0++ - ``images``: Pillow_ 8.3.2 → 9.2.0++ - ``s3``: boto3_ 1.20.0 → 1.35.2, and aioboto3_ >= 9.0.0 is now+ included.++ - ``twisted-http2``: Twisted_ 21.7.0 → 23.10.0++ - ``uvloop``: uvloop_ 0.16.0 → 0.17.0++ (:gh:`6430`, :gh:`7826`, :gh:`7932`, :gh:`8058`, :gh:`8062`, :gh:`8162`,+ :gh:`8213`, :gh:`8261`, :gh:`8330`, :gh:`8341`, :gh:`8460`)+Backward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- Scrapy no longer officially supports PyPy. It may still work if you- manually remove ``PyDispatcher`` and install ``PyPyDispatcher`` but this is- not guaranteed.- (TBD)-- Changes to the ``fetch()`` shortcut of the :ref:`Scrapy shell<topics-shell>`:- Requests are now downloaded with:meth:`ExecutionEngine.download_async()<scrapy.core.engine.ExecutionEngine.download_async>` instead of being- passed to the scheduler. As a result, spider middlewares are no longer- applied to them.+ passed to the scheduler. As a result, spider middlewares and duplicate+ filtering are no longer applied to them.- When a :class:`~scrapy.Request` object is passed, its ``callback`` and``errback`` are no longer called, and a warning is issued if they areset.- (:gh:`8176`)+ - The engine is no longer started, so the :signal:`engine_started` and+ :signal:`engine_stopped` signals are not sent, and components that start+ on them, such as the :class:`~scrapy.extensions.memusage.MemoryUsage`+ extension and the :ref:`telnet console <topics-telnetconsole>`, do not+ run in the shell.++ (:gh:`8174`, :gh:`8176`, :gh:`8179`)- Changes to the behavior of :class:`~scrapy.core.engine.ExecutionEngine`methods that change its state:- :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` now also- stops the engine. Calling it while the spider is closing now completes- immediately instead of waiting for the in-flight requests, and calling it- after the spider has been closed no longer raises :exc:`RuntimeError`.+ stops the engine. Calling it while the spider is opening now closes the+ spider as soon as it is open, calling it while the spider is closing now+ completes immediately instead of waiting for the in-flight requests, and+ calling it after the spider has been closed no longer raises+ :exc:`RuntimeError`.- :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` now just calls:meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` with the@@ -67,7 +154,10 @@ Backward-incompatible changes:exc:`RuntimeError` if called before:meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` hasfinished, instead of sending the :signal:`engine_started` signal- anyway.+ anyway, and if called after the engine has stopped.++ - The :signal:`engine_stopped` signal is now sent only after all+ :signal:`engine_started` handlers have finished.- If the spider is closed before the engine is started, for example bycalling :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`@@ -75,19 +165,26 @@ Backward-incompatible changes:signal:`engine_stopped` signals are not sent.- :meth:`Crawler.stop_async() <scrapy.crawler.Crawler.stop_async>`, when- called while the spider is being opened, now closes the spider as soon- as it is open, instead of being ignored.+ called before the engine is running, e.g. while the spider is being+ opened, now closes the spider as soon as it is open, instead of being+ ignored.- A :exc:`~scrapy.exceptions.CloseSpider` exception raised while the spideris opening now closes the spider as soon as it is open, instead of beingre-raised by:meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`.- (:gh:`6916`, :gh:`8217`, :gh:`8272`)+ - ``crawl()`` and+ :meth:`~scrapy.core.engine.ExecutionEngine.download_async` now raise+ :exc:`RuntimeError` when called while the spider is closing, once the+ close has stopped waiting for the requests in progress.++ (:gh:`6916`, :gh:`8217`, :gh:`8272`, :gh:`8319`)- Setting the :attr:`ExecutionEngine.running<scrapy.core.engine.ExecutionEngine.running>` attribute no longer has any- effect.+ effect, and is deprecated. Its value is now ``False`` while the spider is+ closing.(:gh:`8181`)- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now@@ -97,9 +194,1568 @@ Backward-incompatible changesthat fits them, with :setting:`DOWNLOAD_MAXSIZE` or with the:reqmeta:`download_maxsize` request meta key for specific requests. You canalso set the limit to ``0`` to lift it altogether.+ (:gh:`8213`)++- The built-in HTTP download handlers now limit their open connections with+ the new :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER` setting, whose+ default value is derived from the limit of open files of the process, or+ is 480 on Windows.+ :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and+ :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` had no+ such limit before, and+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+ and :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`+ used :setting:`CONCURRENT_REQUESTS` as their limit. If you set+ :setting:`CONCURRENT_REQUESTS` higher than that default, a warning is+ logged, and you need to set :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER`+ or, on platforms other than Windows, raise the limit of open files of the+ process (see :ref:`faq-too-many-open-files`).+ (:gh:`2653`, :gh:`5024`, :gh:`5618`, :gh:`7932`)++- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no+ longer falls back to the ``httpx`` package when httpx2_ is not installed.+ Install the ``httpx`` :ref:`extra <extras>` to use it.+ (:gh:`7932`)++- Responses that your code keeps in memory, e.g. in the+ :attr:`.Request.meta` of scheduled requests or in component attributes,+ now count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE` until they are+ released, and so do responses in a reference cycle until Python's garbage+ collector frees them. Keeping many responses can now slow down your crawl.+ To find out whether your crawl keeps responses alive, see+ :ref:`topics-leaks`.++ If requests have been held back by the limit for 5 seconds with none being+ downloaded and no response released, Scrapy runs Python's garbage collector+ and, if that releases no response, lets one request through, logging a+ warning the first time and increasing the new+ :stat:`response_max_active_size/bypass_count` stat.++ Scrapy itself no longer keeps responses alive in reference cycles after a+ request fails, e.g. with an+ :exc:`~scrapy.spidermiddlewares.httperror.HttpError` that no errback+ handles, or after a failed media pipeline download.++ (:gh:`6430`, :gh:`8417`, :gh:`8459`, :gh:`8460`)++- Header names in ``Headers`` objects, i.e. in :attr:`Request.headers+ <scrapy.Request.headers>` and :attr:`Response.headers+ <scrapy.http.Response.headers>`, now keep the case they are first set+ with, instead of being converted to ``Title-Case``.+ Lookups are still case-insensitive, but code that iterates over header+ names and compares them to ``Title-Case`` strings must be updated.++ Over HTTP/1.1,+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+ and :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`+ now send header names in the case you write them in, and keep the case of+ response header names as received. See the :ref:`comparison of download+ handlers <download-handlers-ref>`.++ (:gh:`2711`, :gh:`7893`, :gh:`8265`)++- :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, and hence+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`, now sets the+ new :reqmeta:`skip_dupefilter_once` request meta key on retry requests+ instead of setting their :attr:`~scrapy.Request.dont_filter` attribute to+ ``True``. As a result, with schedulers that support that meta key, like+ the default one, requests derived from a retry request, e.g. redirects,+ are no longer exempt from duplicate filtering.+ (:gh:`6895`, :gh:`8224`)++- Requests whose meta is copied from a downloaded request, e.g. redirects and+ retries, are now assigned to the download slot of their own URL domain,+ instead of keeping the slot of the original request. Requests whose meta is+ copied from a request whose :reqmeta:`download_slot` request meta key you+ set keep that slot, so set it to keep redirects and retries in the slot of+ the original request.+ (:gh:`8337`)++- :reqmeta:`http_user` and :reqmeta:`http_pass` are now only sent to the+ origin (scheme, host and port) of the request that you set them on, so+ they are no longer sent after a cross-origin redirect. To send them to a+ different origin, set the :reqmeta:`http_auth_domain` request meta key.+ :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` also no+ longer sets the ``Authorization`` header of requests whose URL scheme is+ neither ``http`` nor ``https``.+ (:gh:`8329`)++- Keys of :ref:`component priority dictionaries+ <component-priority-dictionaries>`, such as+ :setting:`DOWNLOADER_MIDDLEWARES`, that cannot be resolved into a+ component, e.g. the import path of a component that no longer exists, now+ raise :exc:`ValueError`, even if their priority is ``None``. Some of them+ used to be silently ignored, and a missing module used to raise+ :exc:`ModuleNotFoundError` instead. Fix the import path or remove the+ entry.+ (:gh:`7820`, :gh:`8159`)++- :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` now also+ extracts links with the ``data`` and ``s3`` schemes by default, since+ Scrapy can download them. Use its new ``schemes`` parameter, which+ defaults to :data:`scrapy.linkextractors.SUPPORTED_SCHEMES`, to choose the+ URL schemes of extracted links, e.g. ``schemes=["file", "ftp", "http",+ "https"]`` to extract only the schemes that it extracted before.+ (:gh:`4979`, :gh:`7994`)++- :class:`HttpCompressionMiddleware+ <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`+ now raises :exc:`~scrapy.exceptions.DecompressionError` for a response body+ that cannot be decompressed, instead of the exception of the underlying+ decompression library, e.g. :exc:`zlib.error`, :exc:`OSError` or+ :exc:`EOFError`. Update any code that catches those exceptions, e.g. in+ request errbacks.+ (:gh:`8177`)++- ``str()`` of a :class:`~scrapy.Spider` object now returns its+ :attr:`~scrapy.Spider.name`. Use ``repr()`` to get the previous output,+ e.g. ``<MySpider 'my_spider' at 0x7f0e2c1a3b50>``.+ (:gh:`1576`, :gh:`8086`)++- The ``scrapy.utils.defer.parallel()`` and+ ``scrapy.utils.defer.parallel_async()`` functions, now deprecated, return a+ :class:`~twisted.internet.defer.Deferred` that fires with ``None``, instead+ of with the result list of a :class:`~twisted.internet.defer.DeferredList`.+ (:gh:`5182`, :gh:`7959`)++- The ``process_response()`` method of :class:`HttpCompressionMiddleware+ <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`+ is now a coroutine function. Code that calls it must now await it, and+ subclasses that override it and call ``super().process_response()`` must+ now be ``async def``.+ (:gh:`3751`, :gh:`8153`)++- The ``item_scraped()`` method of+ ``scrapy.extensions.feedexport.FeedExporter``, and the+ ``start_exporting()`` and ``finish_exporting()`` methods of+ ``scrapy.extensions.feedexport.FeedSlot``, are now coroutine functions.+ (:gh:`5622`, :gh:`6273`, :gh:`7800`)++- ``parent_id`` is now part of :attr:`Request.attributes+ <scrapy.Request.attributes>`, so :meth:`Request.replace()+ <scrapy.Request.replace>` and+ :func:`~scrapy.utils.request.request_from_dict` pass a ``parent_id``+ keyword argument to the request class. :class:`~scrapy.Request` subclasses+ whose ``__init__()`` method accepts neither ``parent_id`` nor+ ``**kwargs`` must be updated.+ (:gh:`8225`)++- :ref:`Item exporters <topics-exporters>` now turn the values of fields+ without a serializer into lists if they are list-like, e.g. tuples or sets,+ and into dictionaries if they are items, recursively. This changes the+ output of :class:`~scrapy.exporters.PickleItemExporter`,+ :class:`~scrapy.exporters.MarshalItemExporter`,+ :class:`~scrapy.exporters.PprintItemExporter` and custom+ :class:`~scrapy.exporters.BaseItemExporter` subclasses, and+ :class:`~scrapy.exporters.CsvItemExporter` now joins list-like values,+ e.g. sets, the way it joins lists.+ (:gh:`3888`, :gh:`3890`, :gh:`8057`)++- The output of a request :ref:`errback <errbacks>` that runs because a+ download failed now goes through the+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`+ methods of :ref:`spider middlewares <topics-spider-middleware>`, with+ ``None`` as the response, and exceptions raised while iterating that output+ reach their+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`+ methods. Spider middlewares whose methods expect a response must now also+ handle ``None``. Built-in spider middlewares now also process that output,+ e.g. :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` can+ drop requests from such errbacks, and+ :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware` now sets+ :reqmeta:`is_start_request` in its ``process_start()`` method instead of in+ ``get_processed_request()``.+ (:gh:`4999`, :gh:`7992`)++- Feed storages now refuse :ref:`feed modes <feed-mode>` that they do not+ support, which prevents the crawl from starting. In particular, setting+ the ``overwrite`` feed option to ``False`` for an :ref:`S3+ <topics-feed-storage-s3>` feed, which used to log a warning, now raises+ :exc:`ValueError`. For a :ref:`GCS <topics-feed-storage-gcs>` feed, which+ used to log a warning and overwrite the file, it now appends to the file.+ (:gh:`5580`, :gh:`7826`, :gh:`8173`)++- :attr:`~scrapy.spiders.SitemapSpider.sitemap_follow` now also filters the+ sitemap URLs declared in :file:`robots.txt` files. If you set it, make sure+ it matches the URLs of the sitemaps declared in the :file:`robots.txt`+ files that you want to follow.+ (:gh:`1182`, :gh:`8175`)++- The ``process_request()`` method of+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` is now+ a coroutine function. Code that calls it must now await it, and subclasses+ that override it and call ``super().process_request()`` must now be+ ``async def``.+ (:gh:`2744`, :gh:`7915`)++- ``XMLFeedSpider.parse_nodes()`` and ``CSVFeedSpider.parse_rows()`` are now+ asynchronous generators. Code that calls them must now iterate over their+ output with ``async for``.+ (:gh:`8392`)++- The settings of the built-in storages of :ref:`media pipelines+ <topics-media-pipeline>` are no longer set as class attributes of those+ storages, e.g. ``S3FilesStore.POLICY`` keeps its default value regardless+ of :setting:`FILES_STORE_S3_ACL`. Read them from the storage object of the+ pipeline instead, e.g. ``pipeline.store.POLICY``.+ (:gh:`7925`)++- ``Scraper.slot`` and the deprecated ``scrapy.core.scraper.Slot`` class+ now take a :class:`~scrapy.crawler.Crawler` object instead of a maximum+ active size, their ``finish_response()`` method no longer takes a+ *result* parameter, their ``MIN_RESPONSE_SIZE`` attribute was removed,+ and their deprecated ``active_size`` attribute is read-only.+ (:gh:`6430`)++- The engine status that the :ref:`telnet console <topics-telnetconsole>`+ reports now includes ``engine.response_active_size`` instead of+ ``engine.scraper.slot.active_size`` and+ ``engine.scraper.slot.needs_backout()``.+ (:gh:`6430`)++- The deprecated ``METAREFRESH_IGNORE_TAGS`` setting and the *ignore_tags*+ parameter of the deprecated ``scrapy.utils.response.get_meta_refresh()``+ function now raise :exc:`ValueError` for tags other than ``script``,+ ``noscript`` and ``template``, and only whether ``noscript`` is listed has+ any effect. ``METAREFRESH_IGNORE_TAGS`` is ignored altogether if+ :setting:`METAREFRESH_SCRIPTING` is set with the same or a higher+ :ref:`priority <populating-settings>`.+ (:gh:`8393`)++- Scrapy no longer calls the deprecated ``spider_opened()`` and+ ``spider_closed()`` methods of+ :class:`~scrapy.extensions.spiderstate.SpiderState`, so overriding them in+ a subclass has no effect. Handle the :signal:`spider_state_loaded` and+ :signal:`spider_state_saving` signals instead.+ (:gh:`5153`, :gh:`7650`, :gh:`7656`)++- Due to the new minimum version of w3lib_, 2.6.0:++ - Request URLs are now normalized the way web browsers normalize them, so+ :attr:`Request.url <scrapy.Request.url>` and :attr:`Response.url+ <scrapy.http.Response.url>` can differ from the URL you pass, e.g.+ ``https://EXAMPLE.com`` becomes ``https://example.com/``,+ ``https://example.com/a/../b`` becomes ``https://example.com/b``, and+ ``https://0x7f.1/`` becomes ``https://127.0.0.1/``. Update any code that+ compares request or response URLs to literal strings.++ - :ref:`Request fingerprints <request-fingerprints>` might differ for some+ URLs, so that cached responses of+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` and+ the seen requests of a :ref:`job directory <job-dir>` from an earlier+ version might not match the same requests.++ (:gh:`8393`)++- For encodings declared in the response, :attr:`TextResponse.encoding+ <scrapy.http.TextResponse.encoding>` now reports the Python codec names of+ the encodings that web browsers use, e.g. ``gbk`` instead of ``gb18030``+ for a ``gb2312`` label, and ``utf-16-le`` instead of ``utf-16``. It also+ reports ``utf-8`` instead of ``cp1252`` when the response declares no+ encoding and its body is ASCII-only, e.g. empty. For encoding labels that+ web browsers decode as a single U+FFFD character, like ``hz``, it reports+ ``replacement``, which raises :exc:`UnicodeDecodeError` when decoding any+ non-empty body, so use :attr:`TextResponse.text+ <scrapy.http.TextResponse.text>` instead of+ ``response.body.decode(response.encoding)``.++ :attr:`TextResponse.text <scrapy.http.TextResponse.text>` can also differ:++ - The detected encoding of responses that declare none and are not valid+ UTF-8 can change.++ - Encodings that web browsers ignore, like ``utf-32`` or ``cp437``, are now+ ignored when declared, and the encoding is detected instead. A ``<meta>``+ tag or XML declaration that declares UTF-16 now means UTF-8.++ - Encoding declarations in ``<meta>`` tags are now found past the first+ 4096 bytes of the response body.++ - Response bodies are decoded the way web browsers decode them, which for+ some encodings, like ``euc-jp``, differs from Python codecs.++ (:gh:`8393`)++- The base URL of a response, used by :meth:`TextResponse.urljoin()+ <scrapy.http.TextResponse.urljoin>`, :ref:`selectors <topics-selectors>`+ and :ref:`link extractors <topics-link-extractors>`, and the meta refresh+ that :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware`+ follows, are now found closer to the way web browsers find them. In+ particular, they are now found anywhere in the response body, instead of+ only in its first 4096 characters, the meta refresh with the shortest+ delay, and the last one among those with that delay, now wins over the+ first one, meta refreshes within ``<script>`` or ``<template>`` elements,+ or to the response URL itself, ignoring its fragment, are no longer+ followed, and the base URL is normalized like request URLs.+ (:gh:`8393`)++- The ``response`` attribute of :class:`~scrapy.Selector` objects is now+ read-only. For the selector of a response, i.e.+ :attr:`TextResponse.selector <scrapy.http.TextResponse.selector>`, it no+ longer keeps the response alive, and becomes ``None`` once nothing else+ refers to the response.+ (:gh:`8245`)++- A negative value of :setting:`FILES_EXPIRES` or :setting:`IMAGES_EXPIRES`+ now disables expiration, so that files are never downloaded again once+ fetched. Previously, it made files be downloaded again every time; set+ the setting to ``0`` for that.+ (:gh:`8046`)++- Crawls closed because :meth:`~scrapy.Spider.start` raised an exception, or+ by :setting:`CLOSESPIDER_ERRORCOUNT`, now make the :command:`crawl` and+ :command:`runspider` commands exit with code 1.+ (:gh:`8082`, :gh:`8203`)++- A blank line between :ref:`contract <topics-contracts>` lines in a callback+ docstring now starts a new batch of contracts, which needs its own ``@url``+ line; otherwise, the contracts of that batch do not run.++ ``ContractsManager.extract_contracts()`` now returns a list of batches,+ each a list of contracts, and ``ContractsManager.from_method()`` now+ returns a list of requests instead of a request or ``None``.++ (:gh:`768`, :gh:`8043`)++Deprecations+~~~~~~~~~~~~++- The following modules are deprecated, with no replacement:++ - ``scrapy.utils.benchserver``++ - ``scrapy.utils.console``++ - ``scrapy.utils.datatypes``++ - ``scrapy.utils.display``++ - ``scrapy.utils.ftp``++ - ``scrapy.utils.gz``++ - ``scrapy.utils.job``++ - ``scrapy.utils.ossignal``++ - ``scrapy.utils.sitemap``++ - ``scrapy.utils.ssl``++ - ``scrapy.utils.template``++ - ``scrapy.utils.versions``++ (:gh:`7766`, :gh:`8135`, :gh:`8146`, :gh:`8270`)++- The following functions, classes and methods are deprecated, with no+ replacement:++ - ``scrapy.core.scraper.Scraper.handle_spider_error()``++ - ``scrapy.spiders.sitemap.iterloc()``++ - ``scrapy.utils.defer.aiter_errback()``++ - ``scrapy.utils.defer.iter_errback()``++ - ``scrapy.utils.defer.parallel()``++ - ``scrapy.utils.defer.parallel_async()``++ - ``scrapy.utils.python.MutableAsyncChain``+ (:gh:`8175`, :gh:`8288`, :gh:`8350`, :gh:`8363`, :gh:`8414`)++- ``scrapy.utils.python.garbage_collect()`` is deprecated, use+ :func:`gc.collect` instead.+ (:gh:`8367`)++- Passing a coroutine to ``scrapy.utils.spider.iterate_spider_output()``, for+ which it returns a :class:`~twisted.internet.defer.Deferred` instead of an+ iterable, is deprecated. Await the coroutine and pass its result instead.+ (:gh:`8394`)++- Passing a synchronous iterable to+ ``scrapy.core.scraper.Scraper.handle_spider_output_async()`` is deprecated,+ pass an asynchronous iterable instead.+ (:gh:`8288`)++- ``scrapy.utils.reactor.is_asyncio_reactor_installed()`` is deprecated. To+ check for asyncio support, use+ :func:`~scrapy.utils.asyncio.is_asyncio_available`. To check whether a+ reactor is installed, use ``scrapy.utils.reactor.is_reactor_installed()``.+ See :ref:`asyncio-old-versions` for code that must also support older+ Scrapy versions.+ (:gh:`8128`, :gh:`8454`)++- Defining a ``parse_item()`` method in a+ :class:`~scrapy.spiders.XMLFeedSpider` subclass is deprecated. Define+ :meth:`~scrapy.spiders.XMLFeedSpider.parse_node` instead.+ (:gh:`8147`)++- Exceptions that the ``process_response()`` method of a downloader+ middleware raises can now reach the ``process_exception()`` methods of the+ downloader middlewares that have not processed the response yet, so that+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` can retry+ them. Because existing ``process_exception()`` methods may not expect those+ exceptions, e.g. the :exc:`~scrapy.exceptions.IgnoreRequest` raised to drop+ a response, this only happens when the new+ :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` setting is ``True``.+ Its default value, ``False``, is deprecated, and a warning is issued when+ an exception would reach the ``process_exception()`` method of a+ non-Scrapy downloader middleware if the setting were ``True``. Check that+ those methods handle such exceptions as intended, and set the setting to+ ``True``. New projects generated with :command:`startproject` already do.+ (:gh:`7790`, :gh:`8177`)++- ``scrapy.utils.deprecate.create_deprecated_class()`` is deprecated, use+ ``formerly.deprecated_class()`` from formerly_ instead.+ ``scrapy.utils.decorators.deprecated()`` is deprecated, use+ :func:`warnings.deprecated` instead (or ``typing_extensions.deprecated()``+ on Python 3.12 and lower).+ (:gh:`7894`)++- ``scrapy.utils.url.add_http_if_no_scheme()`` is deprecated. Use+ :func:`w3lib.url.add_http_if_no_scheme` instead.+ (:gh:`2186`, :gh:`8189`)++- The following methods of ``Headers`` are deprecated:++ - ``fromkeys()``: pass ``(key, value)`` pairs to ``Headers`` instead.++ - ``has_key()``: use the ``in`` operator instead.++ - ``normkey()`` and ``normvalue()``.++ - ``setlist()``: assign the list instead, e.g. ``headers[key] =+ values``.++ - ``setlistdefault()``: use ``setdefault()`` instead.++ - ``to_unicode_dict()``: use ``to_tuple_list()`` instead.++ (:gh:`8259`, :gh:`8264`)++- The ``spider_closed_callback`` parameter of+ :class:`~scrapy.core.engine.ExecutionEngine` is deprecated. Closing the+ spider now stops the engine, and :meth:`Crawler.crawl_async()+ <scrapy.crawler.Crawler.crawl_async>` completes when that happens.+ (:gh:`6916`, :gh:`8217`)++- Setting the :attr:`ExecutionEngine.running+ <scrapy.core.engine.ExecutionEngine.running>` attribute is deprecated, and+ has no effect.+ (:gh:`8181`)++- Custom :ref:`schedulers <topics-scheduler>` that do not set+ :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`+ to ``True`` are deprecated, including subclasses of the default scheduler+ that override its+ :meth:`~scrapy.core.scheduler.BaseScheduler.enqueue_request` method.+ Scrapy turns :reqmeta:`skip_dupefilter_once` into ``dont_filter=True`` for+ them, with a warning. To update a scheduler,+ handle :reqmeta:`skip_dupefilter_once` in its+ :meth:`~scrapy.core.scheduler.BaseScheduler.enqueue_request` method and+ set+ :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`+ to ``True``.+ (:gh:`6895`, :gh:`8224`)++- The ``overwrite`` feed option is deprecated. Use the new ``mode``+ :ref:`feed option <feed-mode>` instead: ``"overwrite"`` instead of+ ``True``, ``"append"`` instead of ``False``. The ``overwrite`` attribute of+ :class:`~scrapy.extensions.feedexport.FTPFeedStorage` is deprecated as+ well.++ The default value of the new :setting:`FEED_MODE` setting, ``None``, is+ also deprecated: it will change to ``"create"`` in a future version of+ Scrapy, so that existing feed targets are not modified. Set+ :setting:`FEED_MODE`, or the ``mode`` option of every feed, to silence the+ deprecation warning. New projects set :setting:`FEED_MODE` to ``"create"``+ in their :file:`settings.py` file.+ (:gh:`5580`, :gh:`7826`)++- The ``STORE_SCHEMES`` attribute of+ :class:`~scrapy.pipelines.files.FilesPipeline` is deprecated. Use the new+ :setting:`MEDIA_STORAGES` setting instead.+ (:gh:`5991`, :gh:`7925`)++- The ``spider_opened()`` and ``spider_closed()`` methods of+ :class:`~scrapy.extensions.spiderstate.SpiderState` are deprecated. Use the+ new :signal:`spider_state_loaded` and :signal:`spider_state_saving`+ signals instead.+ (:gh:`5153`, :gh:`7650`, :gh:`7656`)++- The *spider* and *to_native_str_type* parameters of+ ``scrapy.robotstxt.decode_robotstxt()`` are deprecated and have no effect.+ (:gh:`8316`)++- Calling :meth:`AsyncCrawlerRunner.crawl()+ <scrapy.crawler.AsyncCrawlerRunner.crawl>` without a running asyncio event+ loop while :setting:`TWISTED_REACTOR_ENABLED` is ``False`` is deprecated.+ Call it from a coroutine that runs in the event loop instead, e.g. one+ passed to :func:`asyncio.run`.+ (:gh:`7201`, :gh:`7681`, :gh:`8327`)++- ``ImagesPipeline.convert_image()`` has a new *image_format* keyword-only+ parameter with the original image format. Overriding the method without+ that parameter, and calling it without it, are deprecated.+ (:gh:`8198`, :gh:`8457`)++- The ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` setting is deprecated. Use+ :setting:`RESPONSE_MAX_ACTIVE_SIZE` instead.++ ``scrapy.core.scraper.Slot`` is deprecated as well, and so are the+ ``active_size`` and ``max_active_size`` attributes and the+ ``needs_backout()`` method of ``Scraper.slot``. Use+ :attr:`ExecutionEngine.response_active_size+ <scrapy.core.engine.ExecutionEngine.response_active_size>`,+ :setting:`RESPONSE_MAX_ACTIVE_SIZE` and+ :meth:`ExecutionEngine.needs_backout()+ <scrapy.core.engine.ExecutionEngine.needs_backout>` instead.++ (:gh:`6430`)++- Using a Twisted reactor other than+ :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` is+ deprecated, be it set through the ``TWISTED_REACTOR`` setting, installed+ by code outside Scrapy, or passed to+ :func:`~scrapy.utils.reactor.install_reactor`. Setting ``TWISTED_REACTOR``+ to ``None`` or ``""`` is deprecated as well. Remove any custom+ ``TWISTED_REACTOR`` setting from your code.++ The ``FORCE_CRAWLER_PROCESS`` setting is deprecated as well.++ (:gh:`8111`, :gh:`8454`)++- Passing an encoding that web browsers do not support, e.g. ``cp437``, to+ the *encoding* parameter of :class:`~scrapy.http.TextResponse` is+ deprecated. To decode a response body with such an encoding, use+ ``response.body.decode(encoding)``.+ (:gh:`8393`)++- ``scrapy.utils.response.get_base_url()`` and+ ``scrapy.utils.response.get_meta_refresh()`` are deprecated. Use the new+ :attr:`TextResponse.base_url <scrapy.http.TextResponse.base_url>` and+ :attr:`TextResponse.meta_refresh <scrapy.http.TextResponse.meta_refresh>`+ attributes instead. When there is no meta refresh, the latter is ``None``+ instead of ``(None, None)``.+ (:gh:`8393`)++- The ``METAREFRESH_IGNORE_TAGS`` setting is deprecated. Use the new+ :setting:`METAREFRESH_SCRIPTING` setting instead.+ (:gh:`8393`)++New features+~~~~~~~~~~~~++- :class:`~scrapy.Request` objects can now be awaited, e.g. in a spider+ callback, to send them and get their response. See :ref:`inline-requests`.+ (:gh:`1144`, :gh:`8207`)++- Added a :setting:`DOWNLOAD_HANDLERS_BY_NAME` setting and a+ :reqmeta:`download_handler` request meta key, to :ref:`choose a download+ handler per request <download-handler-ids>`, e.g. a browser-based handler+ for only some requests. The default :ref:`request fingerprinter+ <request-fingerprints>` takes :reqmeta:`download_handler` into account+ when it does not match the URL scheme.++ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now+ copies :reqmeta:`download_handler` from the request that triggers a+ ``robots.txt`` request, and sets the new :reqmeta:`is_robotstxt_request`+ request meta key on the ``robots.txt`` requests that it sends.++ (:gh:`5510`, :gh:`7885`)++- Added a :reqmeta:`body_file` request meta key, which writes the response+ body into a file as it is downloaded, so that large responses can be+ downloaded without keeping them in memory, with :ref:`download handlers+ that support it <download-handlers-ref>`. Set it to a file path, or to+ ``True`` to write into a new temporary file. Those requests get a+ :class:`~scrapy.http.BodyFileResponse`, and neither they nor their+ responses count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`.+ (:gh:`3880`, :gh:`4620`, :gh:`5044`, :gh:`7987`, :gh:`8466`)++- Sending a shutdown signal, e.g. with Ctrl+C, a second time now triggers a+ fast stop, which drops queued and in-progress downloads instead of waiting+ for them to finish. Sending it a third time forces an unclean shutdown,+ which used to happen on the second signal. Dropped requests are counted in+ the new ``downloader/request_dropped_count`` stat.++ :meth:`Crawler.stop_async() <scrapy.crawler.Crawler.stop_async>`,+ :meth:`CrawlerRunner.stop() <scrapy.crawler.CrawlerRunner.stop>` and+ :meth:`AsyncCrawlerRunner.stop() <scrapy.crawler.AsyncCrawlerRunner.stop>`+ have a new keyword-only ``mode`` parameter that can be ``"graceful"`` (the+ default), ``"fast"`` or ``"force"``, which works as ``"fast"`` unless you+ use :class:`~scrapy.crawler.CrawlerProcess` or+ :class:`~scrapy.crawler.AsyncCrawlerProcess`.+ :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` and+ :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` accept+ ``"graceful"`` and ``"fast"``. A custom :setting:`DOWNLOADER` class needs+ a ``stop()`` coroutine method, returning the number of dropped requests,+ for fast stops to drop its downloads.+ (:gh:`4749`, :gh:`5000`, :gh:`7455`, :gh:`8296`, :gh:`8312`)++- Added the :reqmeta:`delay` request meta key, which holds a request for at+ least the given number of seconds before sending it, and the+ :reqmeta:`delayed` request meta key, which lists the delays that a request+ has been held for.+ (:gh:`802`, :gh:`3835`, :gh:`4980`, :gh:`5015`, :gh:`7873`)++- Added :attr:`TextResponse.base_url <scrapy.http.TextResponse.base_url>` and+ :attr:`TextResponse.meta_refresh <scrapy.http.TextResponse.meta_refresh>`,+ which return the base URL of a response and the+ :class:`~scrapy.http.MetaRefresh` that a web browser with JavaScript+ enabled would follow, and which you can set to override what the response+ body says. When+ :setting:`METAREFRESH_SCRIPTING` is ``False``,+ :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware`+ ignores an overridden :attr:`~scrapy.http.TextResponse.meta_refresh`.+ (:gh:`8393`)++- Added a :setting:`METAREFRESH_SCRIPTING` setting, which you can set to+ ``False`` to follow the meta refresh that a web browser with JavaScript+ disabled would follow.+ (:gh:`8393`)++- Added a :setting:`FEED_MODE` setting and a ``mode`` :ref:`feed option+ <feed-options>`, which set what happens when the target of a feed already+ exists: ``"append"`` to it, ``"overwrite"`` it, or ``"create"``, which+ leaves it untouched, logs an error and increases the+ ``feedexport/conflicts/<storage class>`` stat. Custom feed storages declare+ the modes that they support with the new+ :attr:`~scrapy.extensions.feedexport.FeedStorageProtocol.supported_modes`+ attribute.++ The ``-o`` command-line option also follows :setting:`FEED_MODE`, so in a+ project with :setting:`FEED_MODE` set to ``"create"``, ``-o`` leaves an+ existing file untouched. Use ``-O`` to overwrite it.+ (:gh:`5580`, :gh:`7826`)++- Added :class:`~scrapy.dupefilters.DiskDupeFilter`, a+ :setting:`DUPEFILTER_CLASS` that keeps request fingerprints in an SQLite+ database on disk instead of in memory, to lower memory usage on large+ crawls.+ (:gh:`5307`, :gh:`7952`)++- Added a :setting:`JOBDIR_SYNC_EVERY` setting to write the+ :file:`active.json` file of the :ref:`job directory <job-dir>`, and the+ database of :class:`~scrapy.dupefilters.DiskDupeFilter`, while the crawl+ runs, so that a crawl killed before it can stop cleanly can resume from+ its latest state, provided that :setting:`SCHEDULER_DISK_QUEUE` and+ :setting:`SCHEDULER_START_DISK_QUEUE` use SQLite-based queues. Custom+ :setting:`SCHEDULER_PRIORITY_QUEUE` classes need the new+ :attr:`~scrapy.pqueues.PriorityQueueProtocol.changed` attribute and+ :meth:`~scrapy.pqueues.PriorityQueueProtocol.state` method to support it.++ :file:`active.json` is also now written atomically, so that it cannot be+ left half-written.++ (:gh:`7978`, :gh:`8152`, :gh:`8322`)++- Added :attr:`Request.id <scrapy.Request.id>`, a unique identifier of each+ request, and :attr:`Request.parent_id <scrapy.Request.parent_id>`, the+ :attr:`~scrapy.Request.id` of the request it derives from, which Scrapy+ sets for requests from callbacks and errbacks, from+ :meth:`Response.follow() <scrapy.http.Response.follow>` and from+ :meth:`Request.replace() <scrapy.Request.replace>`, e.g. redirects and+ retries. Use the new *parent_id* parameter of :class:`~scrapy.Request` to+ set it on requests that your code builds on behalf of another request, e.g.+ in a downloader middleware. Both survive serialization, e.g. into the disk+ queues of a :ref:`job directory <job-dir>`.+ (:gh:`8225`)++- Added a :command:`genrequest` command, which prints the Python code of a+ :class:`~scrapy.Request` object equivalent to a cURL command, given as an+ argument or read from the system clipboard with the new ``clipboard``+ :ref:`extra <extras>`.+ (:gh:`3990`, :gh:`3991`, :gh:`8058`)++- Added a :command:`complete` command, which prints a :ref:`shell completion+ <completion>` script for Bash, fish or Zsh. Completion covers the spiders,+ settings and templates of the current project, and custom commands, which+ can provide their own candidates by overriding the new+ :meth:`~scrapy.commands.ScrapyCommand.complete_argument` and+ :meth:`~scrapy.commands.ScrapyCommand.complete_option` methods. It replaces+ the completion scripts that were in the :file:`extras` directory of the+ source code repository.+ (:gh:`6453`, :gh:`7899`)++- Added a :setting:`STICKY_META_KEYS` setting to copy request metadata keys+ from a response into the follow-up requests yielded by its callback,+ handled by the new+ :class:`~scrapy.spidermiddlewares.stickymeta.StickyMetaParamsMiddleware`.++ The new middleware has priority 1000 in+ :setting:`SPIDER_MIDDLEWARES_BASE`, and+ :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`+ moved from 1000 to 999.++ (:gh:`3645`, :gh:`3770`, :gh:`4141`, :gh:`7711`, :gh:`8166`)++- :class:`~scrapy.spiders.Rule` has new ``name`` and ``from_rules``+ parameters, to apply a rule only to responses reached through specific+ other rules.+ (:gh:`982`, :gh:`8066`)++- A callback docstring can now define several batches of :ref:`contracts+ <topics-contracts>`, one per paragraph, and a batch can repeat ``@url`` to+ check the callback against more than one sample request. Custom contracts+ can set the new ``generates_request`` attribute of+ :class:`~scrapy.contracts.Contract` to ``True`` to behave like ``@url``.+ (:gh:`768`, :gh:`8043`)++- The :ref:`Google Cloud Storage feed storage <topics-feed-storage-gcs>` now+ supports the ``"append"`` :ref:`feed mode <feed-mode>`, instead of logging+ a warning and overwriting the feed.+ (:gh:`7823`, :gh:`8173`)++- :setting:`FILES_EXPIRES` and :setting:`IMAGES_EXPIRES` now support+ fractional numbers of days, e.g. ``0.5`` for 12 hours, and negative values,+ which disable expiration, so that files are never downloaded again once+ fetched.+ (:gh:`2309`, :gh:`4731`, :gh:`8046`)++- The *reactor_path* parameter of+ :func:`~scrapy.utils.reactor.install_reactor` is now optional, and+ defaults to the asyncio reactor.+ (:gh:`8125`)++- Added a :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` setting which,+ when ``True``, passes exceptions raised by the ``process_response()``+ method of a downloader middleware to the ``process_exception()`` methods of+ the downloader middlewares that have not processed the response yet. This+ allows :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` to+ retry responses that other middlewares reject, e.g. responses that cannot+ be decompressed, for which :exc:`~scrapy.exceptions.DecompressionError` is+ now part of :setting:`RETRY_EXCEPTIONS`.+ (:gh:`1900`, :gh:`3506`, :gh:`7790`, :gh:`8177`)++- :exc:`~scrapy.exceptions.CloseSpider` can now be raised from+ :ref:`downloader middlewares <topics-downloader-middleware>` to close the+ spider.+ (:gh:`2578`, :gh:`8177`)++- Added :class:`JsonValidationMiddleware+ <scrapy.downloadermiddlewares.jsonvalidation.JsonValidationMiddleware>`,+ with priority 560 in :setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled+ with the :setting:`JSONVALIDATION_ENABLED` setting, which raises+ :exc:`json.JSONDecodeError` for successful+ :class:`~scrapy.http.JsonResponse` responses with an invalid JSON body.+ :exc:`json.JSONDecodeError` is now part of :setting:`RETRY_EXCEPTIONS`, so+ those responses are retried if+ :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` is ``True``.+ (:gh:`8215`)++- Added an *error* keyword-only parameter to+ :exc:`~scrapy.exceptions.CloseSpider` and+ :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`, to mark a+ crawl as failed. Failed crawls set the new :stat:`finish_reason_error` stat+ to ``True``, get ``error=True`` in their :signal:`spider_closed` signal,+ and make the :command:`crawl` and :command:`runspider` commands exit with+ code 1. Crawls closed because :meth:`~scrapy.Spider.start` raised an+ exception (``start_error``) are now also marked as failed, and so are+ crawls closed by :setting:`CLOSESPIDER_ERRORCOUNT`.+ (:gh:`1241`, :gh:`2171`, :gh:`3653`, :gh:`3654`, :gh:`8082`, :gh:`8203`,+ :gh:`8204`, :gh:`8352`, :gh:`8357`)++- Added an :setting:`IMAGES_PRESERVE_FORMAT` setting to :ref:`keep images in+ their original format <preserving-image-format>` instead of converting them+ to JPEG. Stored images then get no file extension.+ (:gh:`1705`, :gh:`2471`, :gh:`8198`)++- Added an ``item_processor`` :ref:`feed option <feed-options>` to+ :ref:`replace each item with any number of items <item-processor>` in a+ given feed, and a ``feedexport/item_count/<storage class>`` stat with the+ number of exported entries per storage class.+ (:gh:`5905`, :gh:`7911`)++- Added an :setting:`UPLOAD_TIMEOUT` setting to set how long uploads to+ remote :ref:`feed storages <topics-feed-storage-backends>` and :ref:`media+ pipeline storages <topics-media-pipeline>` (Amazon S3, Google Cloud Storage+ and FTP) wait for a response before giving up.+ (:gh:`5011`, :gh:`8162`)++- Added a :setting:`MEDIA_CACHE_SIZE` setting to limit how many download+ outcomes :ref:`media pipelines <topics-media-pipeline>` keep in memory,+ which by default they keep for the whole crawl.+ (:gh:`939`, :gh:`8193`)++- :setting:`LOG_FORMAT` now supports a ``%(spider)s`` placeholder with the+ name of the spider that triggered the log message, or ``-`` for messages+ not tied to a spider.+ (:gh:`1576`, :gh:`8086`)++- :class:`~scrapy.statscollectors.StatsCollector` objects are now+ :class:`mutable mappings <collections.abc.MutableMapping>` of stats, e.g.+ ``stats["hostname"] = socket.gethostname()``.+ (:gh:`3230`, :gh:`8241`)++- The :meth:`~scrapy.exporters.BaseItemExporter.start_exporting`,+ :meth:`~scrapy.exporters.BaseItemExporter.export_item` and+ :meth:`~scrapy.exporters.BaseItemExporter.finish_exporting` methods of+ :ref:`item exporters <topics-exporters>` used by :ref:`feed exports+ <topics-feed-exports>`, and the+ :meth:`~scrapy.extensions.feedexport.FeedStorageProtocol.open` and+ :meth:`~scrapy.extensions.feedexport.FeedStorageProtocol.store` methods of+ :ref:`feed storages <topics-feed-storage-custom>`, can now be coroutine+ functions.+ (:gh:`5622`, :gh:`6273`, :gh:`7800`)++- Added a :setting:`LOG_LEVELS` setting, to set the minimum log level of+ specific loggers, overriding the levels that Scrapy sets by default.+ (:gh:`2231`, :gh:`7921`)++- The :command:`settings` command now takes the settings of enabled+ :ref:`add-ons <topics-addons>` into account, and has a new ``--spider``+ option to take the settings of a spider into account as well.+ (:gh:`7010`, :gh:`7828`)++- Added a :reqmeta:`skip_dupefilter_once` request meta key, which skips+ duplicate filtering for a request without affecting the requests derived+ from it, and a+ :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`+ attribute for schedulers to declare that they support it.+ (:gh:`6895`, :gh:`8224`)++- The FTP download handlers now list the contents of a directory when the+ URL path is empty or ends in ``/``, returning the output of the FTP+ ``NLST`` command, one name per line.+ (:gh:`1478`, :gh:`2505`, :gh:`8197`)++- The :ref:`memory usage extension <topics-extensions-ref-memusage>` now+ works on Windows, with the new ``memusage`` :ref:`extra <extras>`.+ (:gh:`4070`, :gh:`8062`)++- Added :class:`~scrapy.core.engine.EngineState` and+ :attr:`ExecutionEngine.state <scrapy.core.engine.ExecutionEngine.state>`,+ which tell the lifecycle state of the engine.+ (:gh:`6916`, :gh:`8217`)++- The :command:`fetch`, :command:`view`, :command:`shell` and+ :command:`parse` commands now support a ``--curl`` option, to build the+ request from a cURL command instead of from a URL argument.+ (:gh:`36`, :gh:`447`, :gh:`7703`)++- Added an experimental FTP and FTPS download handler,+ :class:`~scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler`,+ used by default when running :ref:`without a Twisted reactor+ <asyncio-without-reactor>`. The new ``socks`` :ref:`extra <extras>`+ enables its support for SOCKS proxies.+ (:gh:`7261`, :gh:`8213`, :gh:`8341`)++- Added :setting:`MEDIA_STORAGES` and :setting:`MEDIA_STORAGES_BASE`+ settings, which map URL schemes to the storage classes of the :ref:`media+ pipelines <topics-media-pipeline>`, to support additional schemes or+ replace built-in storages. Storage classes can define a ``from_crawler()``+ class method to read settings, including settings specific to the files or+ images pipeline, or to a pipeline subclass.++ Pipelines no longer share the settings of built-in storages, so that e.g.+ the files and images pipelines of a crawl can use different values for+ :setting:`FILES_STORE_S3_ACL` and :setting:`IMAGES_STORE_S3_ACL`.+ (:gh:`5991`, :gh:`7925`)++- Added :setting:`FILES_CHECKSUM_ALGORITHM` and+ :setting:`IMAGES_CHECKSUM_ALGORITHM` settings, to calculate the checksums+ of :ref:`media pipeline <topics-media-pipeline>` files with a+ :mod:`hashlib` algorithm other than MD5.+ (:gh:`3306`, :gh:`7951`)++- Added :meth:`SitemapSpider.sitemap_request()+ <scrapy.spiders.SitemapSpider.sitemap_request>`, which you can override to+ customize the requests that :class:`~scrapy.spiders.SitemapSpider` builds+ from sitemap entries, e.g. to pass sitemap data to callbacks.+ (:gh:`2565`, :gh:`3543`, :gh:`8175`)++- Added a :meth:`~scrapy.robotstxt.RobotParser.sitemaps` method to+ :class:`~scrapy.robotstxt.RobotParser`, implemented by the+ :ref:`Protego <protego-parser>` and :ref:`Robotexclusionrulesparser+ <rerp-parser>` parsers, which returns the sitemap URLs declared in the+ :file:`robots.txt` file, and a *response* parameter to the+ :signal:`robots_parsed` signal.+ (:gh:`3967`, :gh:`8069`, :gh:`8175`)++- Added the :signal:`spider_state_loaded`, :signal:`spider_state_saving`+ and :signal:`feed_slots_initialized` signals.+ (:gh:`5153`, :gh:`7650`, :gh:`7656`)++- Added an :ref:`event loop lag monitor <event-loop-lag>` extension, enabled+ by default, which logs a warning when code blocks the event loop for longer+ than the new :setting:`EVENT_LOOP_LAG_THRESHOLD` setting, 10 seconds by+ default.+ (:gh:`3197`, :gh:`8093`)++- When :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`+ denies at least one :ref:`start request <start-requests>` and no start+ request can be crawled, the crawl now finishes with the+ ``robotstxt_denied`` :stat:`finish_reason` instead of ``finished``, and an+ error is logged.+ (:gh:`6164`)++- :func:`~scrapy.utils.httpobj.urlparse_cached` now also accepts a URL+ string, and ``scrapy.utils.url.url_is_from_any_domain()``,+ ``url_is_from_spider()`` and ``url_has_any_extension()`` now also accept+ :class:`~scrapy.Request` and :class:`~scrapy.http.Response` objects.+ Built-in components now reuse URL parsing results more, which makes+ crawls faster.+ (:gh:`8247`)++- Added a :reqmeta:`proxy_headers` request meta key, to send headers to the+ proxy set in :reqmeta:`proxy`, e.g. in the ``CONNECT`` request of HTTPS+ URLs.+ (:gh:`3329`, :gh:`7888`)++- Added+ :class:`~scrapy.downloadermiddlewares.uriuserinfo.UriUserInfoMiddleware`,+ enabled by default, which removes the credentials from request URLs like+ ``https://user:[email protected]`` and sets them as :reqmeta:`http_user`+ and :reqmeta:`http_pass`, or as :reqmeta:`ftp_user` and+ :reqmeta:`ftp_password` for FTP URLs. As a result, :attr:`Request.url+ <scrapy.Request.url>` and :attr:`Response.url <scrapy.http.Response.url>`+ no longer include those credentials after that middleware.+ (:gh:`669`, :gh:`851`, :gh:`1466`, :gh:`4474`)++- Added an :setting:`HTTPCACHE_SCOPE` setting, which allows sharing the HTTP+ cache across spiders and, with+ :class:`~scrapy.extensions.httpcache.FilesystemCacheStorage`, splitting it+ by request host instead of by spider.+ (:gh:`5166`, :gh:`5175`, :gh:`7964`)++- Added a :setting:`LOG_ROOT_FORMATTER` setting, which sets the formatter of+ the root log handler. Such formatters implement the new+ :class:`~scrapy.LogRootFormatterProtocol`, and the new+ :class:`~scrapy.LogRootFormatter` is the default one, which you can also+ use in your own log handlers.+ :setting:`LOG_COLOR` also works with custom formatters.+ (:gh:`8336`)++- Added a :setting:`CONNECTION_KEEPALIVE_TIMEOUT` setting, which sets how+ long the built-in HTTP download handlers keep idle connections open for+ reuse. Its default value, 240 seconds, is the timeout that the+ Twisted-based handlers already used; the aiohttp- and httpx-based handlers+ used their library defaults, 15 and 5 seconds respectively.+ (:gh:`8342`)++- Added :meth:`Response.follow_redirect()+ <scrapy.http.Response.follow_redirect>`, which returns a request that+ follows the ``Location`` header of the response.+ (:gh:`8331`)++- When :setting:`DOWNLOAD_FAIL_ON_DATALOSS` or the+ :reqmeta:`download_fail_on_dataloss` request meta key is ``False``,+ hitting the download timeout while receiving the response body now+ returns the part of the response received so far, with the ``dataloss``+ flag, instead of raising :exc:`~scrapy.exceptions.DownloadTimeoutError`,+ with all built-in download handlers except+ :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`.+ (:gh:`3407`, :gh:`3492`, :gh:`8084`, :gh:`8464`)++- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`+ now supports proxies with an IPv6 address, proxies whose host name only+ resolves to IPv6 addresses, given an IPv6-capable+ :setting:`TWISTED_DNS_RESOLVER`, and HTTPS requests to IPv6 addresses+ through a proxy.+ (:gh:`6674`, :gh:`7090`, :gh:`8116`, :gh:`8343`)++- When asyncio support is available and ``aioboto3`` is installed,+ :ref:`S3 feed storage <topics-feed-storage-s3>` uploads files with it,+ instead of running ``boto3`` in a thread. Similarly, the :ref:`S3 storage+ of media pipelines <media-pipelines-s3>` uses ``aiobotocore``, when+ installed, instead of ``botocore``.+ (:gh:`3845`, :gh:`8330`, :gh:`8339`)++- Added a :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER` setting, which limits+ the number of connections, idle ones included, that each download handler+ keeps open. By default, the limit is derived from the maximum number of+ open files allowed for the process, or from the :func:`select.select`+ limit on Windows, to prevent ``Too many open files`` errors (see+ :ref:`faq-too-many-open-files`).+ (:gh:`2653`, :gh:`5024`, :gh:`5618`, :gh:`7932`)++- Added a :setting:`REDIRECT_HTTP_CODES` setting and a+ :reqmeta:`redirect_http_codes` request meta key, which set the response+ status codes whose ``Location`` header+ :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` follows,+ e.g. to follow the ``Location`` header of 201 responses. Media pipelines+ with :setting:`MEDIA_ALLOW_REDIRECTS` enabled leave responses with those+ status codes to+ :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware`.+ (:gh:`1615`, :gh:`7801`)++- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now+ supports ``https://`` proxy URLs for ``https://`` destination URLs.+ (:gh:`5828`, :gh:`7776`)++- When several requests for the same resource are sent concurrently,+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now+ downloads it only once, and, if the response is cached, serves it from the+ cache to the other requests.+ The new :stat:`httpcache/wait` stat counts the requests that waited for+ another one.+ (:gh:`2744`, :gh:`7915`)++- ``XMLFeedSpider.parse_node()``, ``CSVFeedSpider.parse_row()`` and the+ *errback* of :class:`~scrapy.spiders.Rule` can now be defined as coroutine+ functions (``async def``) or asynchronous generator functions.+ (:gh:`8392`)++- ``item_completed()`` in subclasses of ``MediaPipeline`` can now be defined+ as a coroutine function (``async def``), and ``stat_file()`` in+ :ref:`custom media storages <media-pipeline-custom-storage>` can now await+ :mod:`asyncio` code.+ (:gh:`8413`)++- :meth:`BaseSettings.getlist() <scrapy.settings.BaseSettings.getlist>` now+ parses a string that is a JSON list, e.g. ``-s 'MY_LIST=["a,b", "c"]'``, so+ that items that contain commas can be set from the command line.+ (:gh:`8416`)++- Added a :setting:`RESPONSE_ROUGH_SIZE` setting, the+ :attr:`ExecutionEngine.response_active_size+ <scrapy.core.engine.ExecutionEngine.response_active_size>` attribute, and+ the :stat:`request_backout_seconds/concurrency`,+ :stat:`request_backout_seconds/response_max_active_size` and+ :stat:`request_backout_seconds/total` stats. See+ :ref:`crawl-bottlenecks`.+ (:gh:`5302`, :gh:`6206`, :gh:`6430`)++Improvements+~~~~~~~~~~~~++- The encoding of a response that doesn't declare one is now detected the+ way web browsers do: UTF-8 if the body is valid UTF-8, otherwise a guess+ based on the body and on the top-level domain of the response URL, as+ Firefox does. Encoding declarations in ``<meta>`` tags are also found the+ way web browsers find them, and declared encoding names are mapped to+ encodings the way web browsers map them, e.g. ``gb2312`` to GBK instead of+ GB18030.+ (:gh:`8393`)++- With parsel_ 1.12.0+, :class:`~scrapy.Selector` objects created from a+ UTF-8 response now parse the response body as is, instead of decoding it+ first, which saves time and memory.+ (:gh:`5906`, :gh:`8195`, :gh:`8222`, :gh:`8393`)++- Responses are now freed as soon as nothing refers to them, also after+ using their selector or after their callback raises an exception, instead+ of when the garbage collector runs, which lowers peak memory usage.+ (:gh:`2977`, :gh:`8245`)++- The reactor settings :setting:`DNSCACHE_ENABLED`, :setting:`DNSCACHE_SIZE`,+ :setting:`DNS_TIMEOUT`, :setting:`REACTOR_THREADPOOL_MAXSIZE` and+ :setting:`TWISTED_DNS_RESOLVER` can now be :ref:`set from a spider+ <spider-settings>` when using :class:`~scrapy.crawler.CrawlerProcess` or+ :class:`~scrapy.crawler.AsyncCrawlerProcess`. When :ref:`running multiple+ spiders in the same process <run-multiple-spiders>`, the values of the+ first spider are used, and a warning is issued for any later spider that+ defines a different value. The DNS resolver now gets the+ :class:`~scrapy.crawler.Crawler` of that first spider in its+ ``from_crawler()`` method.+ (:gh:`5988`, :gh:`7923`)++- The :command:`runspider` command now accepts file paths without an+ extension, such as the ones of Bash process substitution+ (``scrapy runspider <(…)``).+ (:gh:`4827`, :gh:`8036`)++- Items are now processed by tasks started on demand, instead of by a fixed+ number of tasks, one per :setting:`CONCURRENT_ITEMS`, started for every+ response, so a high :setting:`CONCURRENT_ITEMS` value no longer slows down+ crawls. :setting:`CONCURRENT_ITEMS` values lower than ``1`` now work as+ ``1``.+ (:gh:`5182`, :gh:`7959`)++- Exceptions raised while exporting an item to a :ref:`feed+ <topics-feed-exports>` now include the item in a note.+ (:gh:`2775`, :gh:`8191`)++- When a :class:`~scrapy.http.TextResponse` is created with a :class:`str`+ body and a UTF-8 encoding, that body is now reused as+ :attr:`~scrapy.http.TextResponse.text` instead of being decoded again.+ (:gh:`8220`)++- Improved the performance of link extraction with+ :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`.+ (:gh:`8126`)++- Compressed response bodies of 8 KiB or more are now decompressed in a+ thread, so that other responses can be processed meanwhile.+ (:gh:`3751`, :gh:`8153`)++- The Scrapy version and the versions of :setting:`LOG_VERSIONS` are now+ logged when each crawler is set up, after :ref:`add-ons <topics-addons>`+ and spider settings are applied, so that they can change+ :setting:`BOT_NAME` and :setting:`LOG_VERSIONS`. They are now also logged+ when using :class:`~scrapy.crawler.CrawlerRunner` or+ :class:`~scrapy.crawler.AsyncCrawlerRunner`. Add-ons can now :ref:`log+ the versions of their dependencies <topics-addons>`.+ (:gh:`8246`)++- ``scrapy.pipelines.files.S3FilesStore`` now supports any header that+ botocore_ supports for uploads, and :ref:`media pipelines+ <topics-media-pipeline>` now document how to set custom headers for files+ uploaded to Amazon S3.+ (:gh:`4788`, :gh:`4813`, :gh:`8171`)++- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now+ sends the ``Content-Length`` request header that you set, instead of+ replacing it, with a warning, with one built from the request body.+ (:gh:`4919`, :gh:`7944`)++- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now+ supports :setting:`DOWNLOAD_FAIL_ON_DATALOSS` and the+ :reqmeta:`download_fail_on_dataloss` request meta key: when the setting or+ meta key is ``False``, responses cut short by a lost connection are now+ returned with the ``dataloss`` flag; otherwise, they now fail with+ :exc:`~scrapy.exceptions.ResponseDataLossError` instead of+ :exc:`~scrapy.exceptions.DownloadFailedError`. HTTP/2 data loss is now also+ detected by+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.+ (:gh:`5048`, :gh:`7989`)++- Spiders that request the :file:`robots.txt` file of a host, like+ :class:`~scrapy.spiders.SitemapSpider`, no longer download it twice when+ :setting:`ROBOTSTXT_OBEY` is ``True``:+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now+ parses the response of a request for the :file:`robots.txt` file of a+ host, if it has not downloaded it yet, instead of downloading it again.+ (:gh:`3967`, :gh:`8069`)++- Requests that set the :reqmeta:`download_maxsize` request meta key but not+ :reqmeta:`download_warnsize` no longer log a warning based on+ :setting:`DOWNLOAD_WARNSIZE`.(:gh:`8213`)+- :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+ is now :ref:`lazy <lazy-download-handlers>`.+ (:gh:`8346`)++- Connecting a handler to a built-in signal, where the handler declares an+ argument which the signal never sends, e.g. because of a typo, now issues a+ warning.+ (:gh:`8345`)++- :func:`~scrapy.utils.asyncio.call_later` and the processing of the output+ of spider callbacks now run in the :mod:`contextvars` context that was+ current when they were scheduled also when not using the asyncio reactor,+ so that context variables set by your code, e.g. for tracing or logging,+ are kept.+ (:gh:`8347`)++- Improved the performance of ``scrapy.utils.python.binary_is_text()``,+ which is used to pick the class of responses.+ (:gh:`8362`)++- :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now+ ignores received cookies with a ``__Secure-`` or ``__Host-`` name prefix+ that do not meet the requirements of that prefix, as web browsers do, e.g.+ such cookies received over plain HTTP.+ (:gh:`5431`, :gh:`8431`)++- Improved the performance of :ref:`feed exports <topics-feed-exports>`,+ especially for items with scalar field values.+ (:gh:`8461`)++- Improved the performance of :class:`~scrapy.statscollectors.StatsCollector`+ method calls.+ (:gh:`8243`)++- Other code refactoring and improvements.+ (:gh:`8262`, :gh:`8263`, :gh:`8295`, :gh:`8313`, :gh:`8323`, :gh:`8335`,+ :gh:`8353`, :gh:`8365`, :gh:`8385`, :gh:`8391`, :gh:`8415`)++Bug fixes+~~~~~~~~~++- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now raises+ :exc:`~scrapy.exceptions.DownloadFailedError` when the server drops the+ connection, so that those downloads are retried.+ (:gh:`1511`, :gh:`8206`)++- :class:`~scrapy.pipelines.files.FilesPipeline` now treats responses with+ the 201 status code as successful downloads.+ (:gh:`8349`)++- The ``Cookie`` header now lists cookies with the same path length from+ oldest to newest, as :rfc:`6265#section-5.4` requires.+ (:gh:`5431`, :gh:`8321`)++- The :command:`runspider` command now runs the given file even if its name+ matches that of an already imported module, e.g. :file:`json.py`.+ (:gh:`8036`, :gh:`8463`)++- The :command:`check` command now reports errors that prevent a spider+ from crawling, e.g. errors raised while creating the spider, instead of+ silently ignoring them.+ (:gh:`3716`, :gh:`7991`)++- :meth:`Request.from_curl() <scrapy.Request.from_curl>` now handles+ headers without a value (``-H 'Name;'``) as curl does, ignores headers+ that curl would not send, supports ``-u`` with a user name and no+ password, and raises :exc:`ValueError` for an empty command, instead of+ failing with unexpected exceptions. It also turns ``-H 'Name:'`` into a+ header set to ``None``, i.e. not sent, instead of a header with an empty+ value, accepts Bash ``$'…'`` quoting in any argument, e.g. headers,+ although it does not decode backslash escapes, and keeps a leading ``$`` in+ request data instead of removing it.++ :meth:`Request.to_curl() <scrapy.Request.to_curl>` now includes every+ value of repeated headers, handles headers set to ``None`` and no longer+ fails on headers that are not valid UTF-8.++ (:gh:`8131`, :gh:`8138`, :gh:`8148`, :gh:`8267`, :gh:`8291`, :gh:`8292`)++- ``Headers.to_tuple_list()`` no longer raises :exc:`UnicodeDecodeError`;+ bytes that cannot be decoded are replaced with U+FFFD.+ (:gh:`2592`, :gh:`2637`, :gh:`8192`)++- :func:`copy.copy` on an :class:`~scrapy.Item` now returns a shallow copy,+ like :meth:`Item.copy() <scrapy.Item.copy>`. It used to return an item+ that shared its fields with the original, so that setting or deleting a+ field on one also did it on the other.+ (:gh:`8172`)++- Calling ``exit()`` or ``quit()`` in the :ref:`telnet console+ <topics-telnetconsole>` now closes the connection, instead of logging an+ unhandled error.+ (:gh:`3853`, :gh:`8065`)++- An exception raised while building the :setting:`SCHEDULER` is now+ reported as is, instead of being followed by a+ :exc:`RuntimeError` about an unassigned engine slot.+ (:gh:`8169`)++- The ``lastseen`` field of the representation of a downloader slot, e.g.+ in the :ref:`telnet console <topics-telnetconsole>`, now shows the+ monotonic timestamp of the last activity instead of a meaningless date.+ (:gh:`8182`)++- :class:`~scrapy.spiders.SitemapSpider` now resolves relative and+ protocol-relative URLs in sitemaps against the sitemap URL.+ (:gh:`965`, :gh:`8210`)++- :class:`~scrapy.spiders.CrawlSpider` now follows the links of a response+ also when the callback of the matching rule raises an exception.+ (:gh:`4108`, :gh:`8200`)++- A :ref:`feed <topics-feed-exports>` URI with a placeholder that is neither+ a spider attribute nor returned by the ``uri_params`` function now makes+ only that feed be skipped, with an error logged, instead of raising+ :exc:`KeyError` and leaving the remaining feeds unopened.+ (:gh:`2089`, :gh:`8205`)++- Fixed the file modification times that+ :class:`~scrapy.pipelines.files.FilesPipeline` reads from Amazon S3, Google+ Cloud Storage and FTP storages to apply :setting:`FILES_EXPIRES`. S3 and+ Google Cloud Storage times were off by the UTC offset of the local time+ zone, and FTP times were misread.+ (:gh:`8186`, :gh:`8196`)++- Lines of a callback docstring that start with ``@`` but do not name a+ registered :ref:`contract <topics-contracts>`, e.g. a decorator in a code+ example, are now ignored, instead of raising :exc:`KeyError` or making the+ :command:`check` command treat the callback as having contracts. As a+ result, a misspelled contract name is also ignored.+ (:gh:`8226`)++- The :command:`bench` command now listens on a random free port, instead of+ failing when port 8998 is in use.+ (:gh:`8230`)++- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and the ``concurrency`` key of+ :setting:`DOWNLOAD_SLOTS` are now enforced when using asyncio with no+ download delay.+ (:gh:`8256`, :gh:`8257`)++- :class:`~scrapy.spiders.SitemapSpider` now detects XML sitemaps also in+ responses that are not :class:`~scrapy.http.XmlResponse` objects, e.g.+ from third-party download handlers, based on their headers, URL and body.+ (:gh:`6362`, :gh:`7904`)++- Fixed the handling of headers that are not valid UTF-8 or ASCII in some+ download handlers:++ - :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`+ no longer fails on non-ASCII request headers.++ - :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+ no longer fails on request or response headers that are not valid+ UTF-8. Invalid UTF-8 in request headers is sent as U+FFFD, with a+ warning.++ - :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no+ longer fails, or hangs, on request or response headers that are not+ valid UTF-8.++ (:gh:`8265`)++- ``Headers.get()`` now returns the given default value for a header set to+ ``None``.+ (:gh:`8268`)++- Fixed the ``downloader/request_bytes`` and ``downloader/response_bytes``+ stats, which miscounted line breaks for requests with all headers set to+ ``None`` and for responses with multi-value headers, and+ ``scrapy.utils.request.request_httprepr()``, which added an extra line+ break for requests with all headers set to ``None``.+ (:gh:`8266`, :gh:`8271`)++- Changing the header values in the output of :meth:`Request.to_dict()+ <scrapy.Request.to_dict>` or :meth:`Response.to_dict()+ <scrapy.http.Response.to_dict>` no longer changes the original request or+ response.+ (:gh:`8269`)++- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now+ closes the proxy connection of an HTTPS request that times out, or is+ otherwise cancelled, before the proxy has established the tunnel.+ (:gh:`3614`, :gh:`7810`)++- :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` no+ longer copies the fragment of the original URL into the redirect URL when+ the ``Location`` header ends with an empty fragment (``#``).+ (:gh:`8100`)++- When using a :ref:`job directory <job-dir>`, the default scheduler now pops+ requests by :attr:`~scrapy.Request.priority` across its memory and disk+ queues, instead of popping every request from the memory queue before any+ from the disk queue.++ Custom :setting:`SCHEDULER_PRIORITY_QUEUE` classes need to implement the+ new :meth:`~scrapy.pqueues.PriorityQueueProtocol.next_priority` method to+ get this behavior. :class:`~scrapy.pqueues.PriorityQueueProtocol`+ documents the interface of priority queue classes.+ (:gh:`3666`, :gh:`8237`)++- When resuming a crawl from a :ref:`job directory <job-dir>`, feeds whose+ URI uses ``%(batch_id)d`` now continue the batch numbering of the previous+ run, instead of starting from 1 again and overwriting the existing files.+ (:gh:`5153`, :gh:`7650`, :gh:`7656`)++- :ref:`Item exporters <topics-exporters>` now serialize nested items, and+ lists of them, using the serializers declared in their own fields.+ (:gh:`3888`, :gh:`3890`, :gh:`8057`)++- Fixed :class:`~scrapy.utils.asyncio.AsyncioLoopingCall` sometimes calling+ its function twice in a row when the event loop woke it up slightly early,+ which could, for example, close a spider because of+ :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM` while it was still producing items.+ (:gh:`8294`)++- The :ref:`RobotFileParser <python-robotfileparser>` parser now ignores+ the invalid bytes of a :file:`robots.txt` file that is not valid UTF-8,+ like the other built-in parsers, instead of ignoring the whole file.+ (:gh:`8316`)++- Exceptions raised while handling the output of a spider callback, e.g.+ by a custom scheduler or log formatter, are now logged, and no longer+ interfere with the handling of the rest of that output.+ (:gh:`8318`)++- Scrapy now uses the event loop of the installed asyncio reactor when no+ event loop is running, instead of the current event loop of the thread,+ which can be a different one or none at all, e.g. when the reactor was+ installed with a loop that is not set as the current one.+ (:gh:`7201`, :gh:`8324`, :gh:`8325`)++- Stopping a crawl while an item yielded from :meth:`~scrapy.Spider.start` is+ going through the :ref:`item pipelines <topics-item-pipeline>` no longer+ hangs.+ (:gh:`8348`)++- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now+ fails requests with the connection error when it cannot connect to the+ server, instead of letting them hit the download timeout, and later+ requests to the same server try to connect again.+ (:gh:`8332`)++- Fixed :class:`~scrapy.pqueues.ScrapyPriorityQueue`, which the default+ :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` uses for each+ download slot, returning no request or a request other than the one that+ its ``peek()`` method returned, which could happen with requests of the+ same priority in its start request and regular request queues, or after a+ request failed to be serialized into a disk queue.+ (:gh:`7837`)++- The :ref:`telnet console <topics-telnetconsole>`, and any other caller of+ ``scrapy.utils.reactor.listen_tcp()``, now tries the next port when another+ process starts listening on the same port at the same time, instead of+ failing to start.+ (:gh:`8356`)++- Fixed :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`+ letting through requests disallowed by :file:`robots.txt` when they were+ sent while the :file:`robots.txt` file of their website was being+ downloaded for another request.+ (:gh:`8297`, :gh:`8386`)++- :class:`~scrapy.spiders.SitemapSpider` now only treats links with+ ``rel="alternate"`` as alternate links, instead of any link, e.g.+ canonical ones, both when+ :attr:`~scrapy.spiders.SitemapSpider.sitemap_alternate_links` is enabled+ and in the entries passed to+ :meth:`~scrapy.spiders.SitemapSpider.sitemap_filter`.+ (:gh:`8390`)++- Download handlers are now closed when the engine is closed without having+ been started, e.g. when opening the spider fails, and the downloader is now+ closed when the engine is stopped without a spider.+ (:gh:`8167`, :gh:`8377`)++- The spider is no longer considered idle, and so no longer closed, while a+ request taken from the scheduler has not reached the downloader yet, which+ could happen with a custom :setting:`DOWNLOADER`.+ (:gh:`8315`, :gh:`8355`)++- The reactorless :command:`shell` now waits for the engine to start before+ letting you use it.+ (:gh:`8168`)++- On Windows,+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+ no longer connects to the wrong port in rare cases.+ (:gh:`8382`)++- The Scrapy :command:`shell` no longer uses ptpython_ on Windows when the+ standard output is not a console, which ptpython_ does not support.+ (:gh:`8367`)++Documentation+~~~~~~~~~~~~~++- Documented :ref:`how blocking code in callbacks and other components+ slows down downloads <optimize-blocking>` and inflates their+ :reqmeta:`download_latency`.+ (:gh:`3197`, :gh:`8160`)++- Documented :ref:`how to change concurrency and delays from the telnet+ console <telnet-concurrency>` on a running crawl.+ (:gh:`4196`, :gh:`8164`)++- Documented that the ``open_spider()`` and ``close_spider()`` methods of+ :ref:`item pipelines <topics-item-pipeline>` run concurrently, and only+ ``process_item()`` follows the :setting:`ITEM_PIPELINES` order.+ (:gh:`2645`, :gh:`8158`)++- Documented :ref:`how to set up a development environment <dev-setup>` to+ contribute to Scrapy.+ (:gh:`4451`, :gh:`8165`)++- Moved the release notes of Scrapy 2.13 and older into separate pages per+ major version.+ (:gh:`8150`)++- Documented :ref:`how to extract clean content and parse values+ <extraction>`, :ref:`how to sign requests with OAuth <mw-oauth>`, how to+ deduplicate repeated query string parameters in :ref:`request fingerprints+ <custom-request-fingerprinter>`, and that retried status codes should also+ be in :setting:`HTTPCACHE_IGNORE_HTTP_CODES`.+ (:gh:`40`, :gh:`1113`, :gh:`2297`, :gh:`8190`, :gh:`8202`, :gh:`8208`,+ :gh:`8209`)++- Improved the API reference of :class:`~scrapy.spiders.SitemapSpider`,+ :class:`~scrapy.spiders.XMLFeedSpider` and+ :class:`~scrapy.spiders.CSVFeedSpider`.+ (:gh:`8229`, :gh:`8239`)++- Documented the order of the :signal:`spider_opened`,+ :signal:`engine_started`, :signal:`spider_closed` and+ :signal:`engine_stopped` signals.+ (:gh:`1523`, :gh:`8101`)++- Documented that the :setting:`JOBDIR` and :setting:`HTTPCACHE_DIR`+ directories must be protected from :ref:`write access by untrusted+ parties <security-local-data>`.+ (:gh:`8252`)++- Documented :ref:`how to write custom feed storage backends+ <topics-feed-storage-custom>` and the :ref:`built-in feed storage backend+ classes <feed-storage-classes>`.+ (:gh:`7800`)++- Added an introduction video to the :ref:`page about coding agents+ <agents>`.+ (:gh:`8253`)++- :class:`~scrapy.spiders.CrawlSpider` and :class:`~scrapy.spiders.Rule` are+ now documented through their docstrings, and the documentation now+ recommends :class:`~scrapy.Spider` over+ :class:`~scrapy.spiders.CrawlSpider` for crawls that do not fit its+ declarative rules.+ (:gh:`7025`, :gh:`8240`)++- Documented the+ :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`,+ :meth:`~scrapy.core.engine.ExecutionEngine.start_async`,+ :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` and+ :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` methods of the+ execution engine.+ (:gh:`6164`, :gh:`8462`)++- Documented :ref:`how to write asyncio code that also supports older Scrapy+ versions <asyncio-old-versions>`, and the documentation now recommends+ :func:`asyncio.to_thread` to call blocking code from :ref:`coroutines+ <topics-coroutines>`.+ (:gh:`7956`, :gh:`8454`)++- Reviewed the documentation of :class:`~scrapy.Spider`, which now covers+ all its attributes, including :attr:`~scrapy.Spider.state`.+ (:gh:`7025`, :gh:`7811`)++- Updated the :ref:`installation guide <intro-install>`, and documented+ :ref:`what to do when installing a dependency fails <install-no-wheel>`.+ (:gh:`5308`, :gh:`7878`)++- The :ref:`tutorial <intro-tutorial>` now shows how to crawl+ ``quotes.toscrape.com`` faster with :setting:`DOWNLOAD_SLOTS`.+ (:gh:`8334`)++- Documented :exc:`~scrapy.spidermiddlewares.httperror.HttpError`, the+ IPv6 support of the built-in :ref:`download handlers+ <topics-download-handlers>`, the ``bootstrap_failed`` attribute of crawler+ processes, and where exceptions that interrupt a crawl started from a+ :ref:`script <run-from-script>` reach your code.+ (:gh:`8338`, :gh:`8340`, :gh:`8344`)++- Added a :ref:`page about getting help <getting-help>`.+ (:gh:`3688`)++- Documented that high values of :setting:`CONCURRENT_REQUESTS` may exceed+ the limit of open file descriptors that the operating system sets, and how+ to raise that limit.+ (:gh:`5259`, :gh:`5272`)++- Other documentation improvements and fixes.+ (:gh:`5859`, :gh:`7114`, :gh:`8151`, :gh:`8161`, :gh:`8314`, :gh:`8403`,+ :gh:`8404`, :gh:`8436`, :gh:`8453`)++Quality assurance+~~~~~~~~~~~~~~~~~++- Improved and fixed type hints.+ (:gh:`7811`, :gh:`8078`, :gh:`8183`, :gh:`8311`, :gh:`8333`, :gh:`8440`)++- Improved CodSpeed benchmarks.+ (:gh:`8227`, :gh:`8234`, :gh:`8456`)++- Improved the documentation build and the contributor tooling, including+ an ``AGENTS.md`` file for coding agents.+ (:gh:`8011`, :gh:`8154`, :gh:`8157`, :gh:`8188`, :gh:`8212`, :gh:`8214`,+ :gh:`8216`, :gh:`8219`, :gh:`8238`, :gh:`8251`, :gh:`8364`, :gh:`8384`,+ :gh:`8409`, :gh:`8411`)++- CI and test improvements and fixes.+ (:gh:`6141`, :gh:`7949`, :gh:`8127`, :gh:`8155`, :gh:`8156`, :gh:`8194`,+ :gh:`8223`, :gh:`8228`, :gh:`8231`, :gh:`8232`, :gh:`8233`, :gh:`8235`,+ :gh:`8236`, :gh:`8258`, :gh:`8273`, :gh:`8274`, :gh:`8275`, :gh:`8276`,+ :gh:`8277`, :gh:`8278`, :gh:`8279`, :gh:`8280`, :gh:`8281`, :gh:`8286`,+ :gh:`8300`, :gh:`8301`, :gh:`8311`, :gh:`8314`, :gh:`8317`, :gh:`8320`,+ :gh:`8351`, :gh:`8359`, :gh:`8360`, :gh:`8361`, :gh:`8372`, :gh:`8374`,+ :gh:`8375`, :gh:`8381`, :gh:`8387`, :gh:`8388`, :gh:`8389`, :gh:`8406`,+ :gh:`8412`, :gh:`8423`, :gh:`8444`)+.. _release-2.19.0:Scrapy 2.19.0 (2026-09-10)@@ -3020,17 +4676,33 @@ Older releasesnews/0.x+.. _aioboto3: https://github.com/terricain/aioboto3+.. _aiodns: https://github.com/aio-libs/aiodns+.. _aioftp: https://github.com/aio-libs/aioftp.. _aiohttp: https://docs.aiohttp.org/en/stable/+.. _boto3: https://github.com/boto/boto3.. _botocore: https://github.com/boto/botocore.. _charset-normalizer: https://charset-normalizer.readthedocs.io/en/latest/.. _cryptography: https://cryptography.io/en/latest/+.. _cssselect: https://github.com/scrapy/cssselect+.. _formerly: https://github.com/scrapy/formerly+.. _frostwork: https://github.com/scrapy/frostwork+.. _google-cloud-storage: https://github.com/googleapis/python-storage+.. _httpx2: https://github.com/pydantic/httpx2.. _ipdb: https://github.com/gotcha/ipdb+.. _itemloaders: https://github.com/scrapy/itemloaders.. _lxml: https://lxml.de/+.. _parsel: https://github.com/scrapy/parsel.. _parso: https://github.com/davidhalter/parso.. _platformdirs: https://platformdirs.readthedocs.io/en/latest/.. _Pillow: https://github.com/python-pillow/Pillow+.. _Protego: https://github.com/scrapy/protego+.. _ptpython: https://github.com/prompt-toolkit/ptpython.. _pyOpenSSL: https://www.pyopenssl.org/en/stable/.. _queuelib: https://github.com/scrapy/queuelib.. _service_identity: https://service-identity.readthedocs.io/en/stable/+.. _Twisted: https://twisted.org/+.. _typing-extensions: https://github.com/python/typing_extensions+.. _uvloop: https://github.com/MagicStack/uvloop.. _w3lib: https://github.com/scrapy/w3lib.. _zope.interface: https://zopeinterface.readthedocs.io/en/latest/diff --git a/docs/topics/api.rst b/docs/topics/api.rstindex a96e7f36d..d96e449f3 100644--- a/docs/topics/api.rst+++ b/docs/topics/api.rst@@ -209,7 +209,8 @@ Engine API.. autoclass:: scrapy.core.engine.ExecutionEngine():members: close_spider_async, download_async, needs_backout,- response_active_size, running, scheduler, state+ open_spider_async, response_active_size, running, scheduler, start_async,+ state, stop_async.. autoclass:: scrapy.core.engine.EngineState:members:diff --git a/docs/topics/commands.rst b/docs/topics/commands.rstindex ca221b936..e2e43d0d2 100644--- a/docs/topics/commands.rst+++ b/docs/topics/commands.rst@@ -467,7 +467,7 @@ Supported options:command (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). It cannotbe combined with a URL argument.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Usage examples::@@ -514,7 +514,7 @@ Supported options:URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). Itcannot be combined with a URL argument.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Usage example::@@ -548,7 +548,7 @@ Supported options:URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). Itcannot be combined with a URL argument.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Usage example::@@ -592,7 +592,7 @@ Supported options:URL argument (see :meth:`Request.from_curl() <scrapy.Request.from_curl>`). Itcannot be combined with a URL argument.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0* ``--meta`` or ``-m``: additional request meta that will be passed to thecallback request. This must be a valid JSON string. Example:@@ -667,7 +667,7 @@ a spider with ``--spider``::$ scrapy settings --get DOWNLOAD_DELAY --spider myspider2.0-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0Add-on settings are now taken into account, and the ``--spider`` option wasadded.@@ -707,7 +707,7 @@ genrequest* Syntax: ``scrapy genrequest [curl command]``* Requires project: *no*-.. versionadded:: VERSION+.. versionadded:: 2.20.0Print the Python code of a :class:`~scrapy.Request` object equivalent to a`cURL <https://curl.se/>`_ command. If the curl command is not given as an@@ -738,7 +738,7 @@ complete* Syntax: ``scrapy complete <bash|fish|zsh>``* Requires project: *no*-.. versionadded:: VERSION+.. versionadded:: 2.20.0Print a shell completion script. See :ref:`completion`.@@ -747,7 +747,7 @@ Print a shell completion script. See :ref:`completion`.Shell completion================-.. versionadded:: VERSION+.. versionadded:: 2.20.0The :command:`complete` command prints a completion script for Bash, fish orZsh.diff --git a/docs/topics/contracts.rst b/docs/topics/contracts.rstindex b18b2916b..75b4c06c3 100644--- a/docs/topics/contracts.rst+++ b/docs/topics/contracts.rst@@ -34,7 +34,7 @@ A callback docstring can also be split into several batches of contracts, oneper paragraph separated by a blank line, and a batch may repeat ``@url`` tocheck the callback against more than one sample request:-.. versionadded:: VERSION+.. versionadded:: 2.20.0.. code-block:: pythondiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 58974c22a..0b17233f0 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -63,7 +63,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :meth:`~scrapy.exporters.BaseItemExporter.finish_exporting`- .. versionadded:: VERSION+ .. versionadded:: 2.20.0- Methods of :ref:`custom feed storages <topics-feed-storage-custom>` (see:setting:`FEED_STORAGES`):@@ -72,7 +72,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :meth:`~scrapy.extensions.feedexport.FeedStorageProtocol.store`- .. versionadded:: VERSION+ .. versionadded:: 2.20.0- The following methods of subclasses of:class:`scrapy.pipelines.media.MediaPipeline`:@@ -92,7 +92,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- ``item_completed()``- .. versionadded:: VERSION+ .. versionadded:: 2.20.0- The following methods of :ref:`custom media storages<media-pipeline-custom-storage>`:@@ -103,12 +103,12 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- ``stat_file()``- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Most callables above may also :ref:`await a request <inline-requests>` to sendit and get its response.-.. versionadded:: VERSION+.. versionadded:: 2.20.0.. _coroutine-deferred-apis:@@ -264,7 +264,7 @@ Inline requestsThe spider below shows how to send a request and await its response all fromwithin a spider callback:-.. versionadded:: VERSION+.. versionadded:: 2.20.0Awaiting a :class:`~scrapy.Request` object... code-block:: pythondiff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex e3bb83a71..ea2e2aa77 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -48,7 +48,7 @@ for additional schemes and to replace or disable default ones:Choosing a download handler per request----------------------------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0The :setting:`DOWNLOAD_HANDLERS_BY_NAME` setting registers handlers under aname instead of a URL scheme. Such handlers are only used by requests that ask@@ -245,7 +245,7 @@ AiohttpDownloadHandler.. versionadded:: 2.19.0-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0In earlier versions, the handler was not lazy... autoclass:: scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler@@ -479,7 +479,7 @@ Built-in non-HTTP download handlers referenceAioftpDownloadHandler----------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0.. autoclass:: scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandlerdiff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex a85b4e455..a7e0cb489 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -646,7 +646,7 @@ HTTPCACHE_SCOPEDefault: ``'spider'``-.. versionadded:: VERSION+.. versionadded:: 2.20.0How to partition the cache within :setting:`HTTPCACHE_DIR`:@@ -941,7 +941,7 @@ key set to True, the request will be ignored by this middleware... reqmeta:: redirect_http_codes-.. versionadded:: VERSION+.. versionadded:: 2.20.0The ``redirect_http_codes`` key of :attr:`Request.meta <scrapy.Request.meta>`overrides the :setting:`REDIRECT_HTTP_CODES` setting for a single request.@@ -1016,7 +1016,7 @@ Whether the Redirect middleware will be enabled.REDIRECT_HTTP_CODES^^^^^^^^^^^^^^^^^^^-.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``[301, 302, 303, 307, 308]``@@ -1113,7 +1113,7 @@ restrict automatic redirection to the maximum delay.METAREFRESH_SCRIPTING^^^^^^^^^^^^^^^^^^^^^-.. versionadded:: VERSION+.. versionadded:: 2.20.0.. setting:: METAREFRESH_SCRIPTINGdiff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rstindex b87a99fbe..fcf6c4276 100644--- a/docs/topics/exporters.rst+++ b/docs/topics/exporters.rst@@ -247,7 +247,7 @@ BaseItemExportermethod (in your custom Item Exporters) if you want to control how aparticular field or value will be serialized/exported.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Nested items, and lists of them, are now serialized recursivelyusing their own declared serializers, instead of unchanged.diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex a2ccbe7d7..87b8bcd9e 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -298,7 +298,7 @@ Supported :ref:`modes <feed-mode>`: ``"append"``, ``"create"`` andAppending turns the feed into a `composite object`_, which has a CRC32Cchecksum but no MD5 hash.-.. versionadded:: VERSION+.. versionadded:: 2.20.0Appending support.This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.@@ -357,7 +357,7 @@ URI scheme through the :setting:`FEED_STORAGES` setting.Optional set of supported values of the ``mode`` :ref:`feed option<feed-options>`, e.g. ``frozenset({"create", "overwrite"})``.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Setting ``mode`` to an unsupported value raises an exception, whichprevents the crawl from running. A storage class that does not define@@ -427,7 +427,7 @@ ItemFilterItem processing===============-.. versionadded:: VERSION+.. versionadded:: 2.20.0The ``item_processor`` :ref:`feed option <feed-options>` takes a callable, orits import path, that receives an accepted item and returns an iterable of the@@ -447,8 +447,8 @@ the feed that declares them. Items are exported as returned, so the:signal:`item_scraped` signal, :ref:`item pipelines <topics-item-pipeline>` andthe ``item_scraped_count`` stat still see the item as scraped; use itemprocessors for output formatting, and item pipelines for anything that should-apply to the item itself. The number of exported entries per feed is reported-as the ``feedexport/item_count/<storage class>`` stat.+apply to the item itself. The number of exported entries is reported per+storage class, as the ``feedexport/item_count/<storage class>`` stat... _post-processing:@@ -608,7 +608,7 @@ as a fallback value if that key is not provided for a specific feed definition:- ``item_processor``: an :ref:`item processor <item-processor>` to reshapeitems before they are exported.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0If undefined, items are exported as scraped.@@ -618,7 +618,7 @@ as a fallback value if that key is not provided for a specific feed definition:- ``mode``: falls back to :setting:`FEED_MODE`.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0- ``store_empty``: falls back to :setting:`FEED_STORE_EMPTY`.@@ -675,7 +675,7 @@ to ``.json`` or ``.xml``.FEED_MODE----------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``"create"`` (:ref:`fallback <default-settings>`: ``None``)diff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex bbe443c3f..966826263 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -196,7 +196,7 @@ and strptime directives <strftime-strptime-behavior>` respectively.LOG_ROOT_FORMATTER~~~~~~~~~~~~~~~~~~-.. versionadded:: VERSION+.. versionadded:: 2.20.0.. setting:: LOG_ROOT_FORMATTERdiff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex f70d52111..68fcfe7b0 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -407,7 +407,7 @@ For more information, see `Predefined ACLs`_ in the Google Cloud Platform DeveloCustom storage---------------.. versionadded:: VERSION+.. versionadded:: 2.20.0.. setting:: MEDIA_STORAGES@@ -539,7 +539,7 @@ adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), whichspecifies the delay in number of days:-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0Fractional numbers of days are now supported... code-block:: python@@ -570,7 +570,7 @@ which is then compared to the set expiration time to determine if the file is exChecksum algorithm-------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0.. setting:: FILES_CHECKSUM_ALGORITHM.. setting:: IMAGES_CHECKSUM_ALGORITHM@@ -701,7 +701,7 @@ To handle media redirections, set this setting to ``True``:Media files created on the fly-------------------------------.. versionchanged:: VERSION+.. versionchanged:: 2.20.0Added support for HTTP 201 responses.Some servers create media files on the fly and report that with an HTTP 201@@ -730,7 +730,7 @@ that a file referenced by several items is downloaded once. By default everyoutcome is kept for the whole crawl, which takes about 600 bytes per distinctmedia URL.-.. versionadded:: VERSION+.. versionadded:: 2.20.0To cap that memory, set :setting:`MEDIA_CACHE_SIZE` to the number of mostrecently used outcomes to keep, or to ``0`` to keep none:diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 58a0d8a39..f481d6e31 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -48,7 +48,7 @@ Request objectsA dictionary-like (:class:`scrapy.http.headers.Headers`) object which containsthe request headers.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Header names keep the case you write them in, instead of beingconverted to ``Title-Case``.@@ -900,7 +900,7 @@ the :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported by them.body_file----------.. versionadded:: VERSION+.. versionadded:: 2.20.0Path of a file where the response body is written as it is downloaded, or``True`` to write it to a new temporary file, so that responses of any size can@@ -933,7 +933,7 @@ ones <download-handlers-ref>`, make the request fail withdelay------.. versionadded:: VERSION+.. versionadded:: 2.20.0Number of seconds to hold a request before sending it:@@ -955,7 +955,7 @@ key again asks for a new delay. See also :reqmeta:`delayed`.delayed--------.. versionadded:: VERSION+.. versionadded:: 2.20.0Delays, in seconds, that a request has been held for because of:reqmeta:`delay`. Read-only.@@ -971,7 +971,7 @@ Delays, in seconds, that a request has been held for because ofdownload_handler-----------------.. versionadded:: VERSION+.. versionadded:: 2.20.0ID of the :ref:`download handler <topics-download-handlers>` to use for thisrequest, either a name from :setting:`DOWNLOAD_HANDLERS_BY_NAME` or a URL@@ -1060,7 +1060,7 @@ Overrides :setting:`HTTPAUTH_USER` for this request.is_robotstxt_request---------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0``True`` on the ``robots.txt`` requests that:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` sends, so@@ -1081,7 +1081,7 @@ The meta key is used set retry times per request. When set, theproxy_headers--------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Headers for the proxy set in the :reqmeta:`proxy` metadata key, as a mapping,e.g. ``{"X-Proxy-Country": "us"}``.@@ -1116,7 +1116,7 @@ the proxy to consume them instead of passing them on to the target server.skip_dupefilter_once---------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Set this key to ``True`` to skip :setting:`duplicate filtering<DUPEFILTER_CLASS>` for this request. The :ref:`scheduler <topics-scheduler>`diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 41fa28609..ea9e445d6 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -264,7 +264,7 @@ A key that cannot be resolved into a component, such as the import path of acomponent that no longer exists, raises an exception, even if its priority is:data:`None`.-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0Unresolvable keys used to be silently ignored in some cases... warning:: Component priority dictionaries are regular :class:`dict` objects.@@ -334,7 +334,7 @@ Reactor settings**Reactor settings** are settings tied to the :doc:`Twisted reactor<twisted:core/howto/reactor-basics>`.-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0:setting:`TWISTED_DNS_RESOLVER`, the settings of the resolver and:setting:`REACTOR_THREADPOOL_MAXSIZE` are now read from the first spider,instead of being read from the project settings and ignored in@@ -574,7 +574,7 @@ CONCURRENT_CONNECTIONS_PER_HANDLERDefault: ``None``-.. versionadded:: VERSION+.. versionadded:: 2.20.0The maximum number of connections that a single :ref:`download handler<topics-download-handlers>` may keep open, whether they are being used for a@@ -651,7 +651,7 @@ It is possible to change this setting per domain by usingCONNECTION_KEEPALIVE_TIMEOUT-----------------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``240.0``@@ -995,7 +995,7 @@ should never modify this setting in your project, modifyDOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS------------------------------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``False``@@ -1192,7 +1192,7 @@ handler (without replacement), place this in your ``settings.py``:DOWNLOAD_HANDLERS_BY_NAME--------------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``{}``@@ -1234,7 +1234,7 @@ Redirects and retries are assigned to a slot based on their own URL domain,unless the slot of their original request was set through the ``download_slot``key, in which case they keep that slot.-.. versionchanged:: VERSION+.. versionchanged:: 2.20.0In earlier versions, redirects and retries always kept the slot of theiroriginal request, even when they pointed to a different domain.@@ -1719,7 +1719,7 @@ A string indicating the directory for storing the state of a crawl whenJOBDIR_SYNC_EVERY------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``0``@@ -1844,7 +1844,7 @@ LOG_LEVELSDefault: ``{}``-.. versionadded:: VERSION+.. versionadded:: 2.20.0Minimum level to log for specific loggers.@@ -1855,6 +1855,7 @@ It takes precedence over the levels that Scrapy sets by default::"hpack": "ERROR","httpcore": "ERROR","httpx": "WARNING",+ "parso": "ERROR","scrapy": "DEBUG","twisted": "ERROR",}@@ -2038,7 +2039,7 @@ Adjust redirect request priority relative to original request:RESPONSE_MAX_ACTIVE_SIZE-------------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``5_000_000``@@ -2091,7 +2092,7 @@ stat.RESPONSE_ROUGH_SIZE--------------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``None``@@ -2411,6 +2412,8 @@ For more info see: :ref:`topics-stats`.STICKY_META_KEYS----------------+.. versionadded:: 2.20.0+Default: ``[]`` (empty list)The :attr:`Request.meta <scrapy.http.Request.meta>` keys to copy automatically@@ -2547,7 +2550,7 @@ See :ref:`asyncio-without-reactor` for more information about this mode.UPLOAD_TIMEOUT---------------.. versionadded:: VERSION+.. versionadded:: 2.20.0Default: ``None``diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 41e6fc193..b0c3db5d7 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -273,7 +273,7 @@ spider_closedSent after a spider has been closed. This can be used to release per-spiderresources reserved on :signal:`spider_opened`.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Added the *error* argument.This signal supports :ref:`asynchronous handlers <signal-deferred>`.@@ -611,7 +611,7 @@ robots_parsed.. versionadded:: 2.18.0- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Added *response*.Sent bydiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 4efb0197f..bf87a9049 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -122,7 +122,7 @@ one or more of these methods:This method is an :term:`asynchronous generator` called with theresults from the spider after the spider has processed the response.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0It is also called with the output of a request :ref:`errback<errbacks>`, with *response* set to ``None``, when the errback runsbecause a download failed.@@ -159,7 +159,7 @@ one or more of these methods:<errbacks>` or a :meth:`process_spider_output` method (from a previousspider middleware) raises an exception.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0It is also called for exceptions raised while iterating the outputof an errback that runs because a download failed, with *response*set to ``None``.diff --git a/docs/topics/stats.rst b/docs/topics/stats.rstindex 3dfcfb18c..a74b9824b 100644--- a/docs/topics/stats.rst+++ b/docs/topics/stats.rst@@ -85,7 +85,7 @@ Get all stats:Stats collectors that subclass :class:`StatsCollector`, like the built-in ones,are also :class:`mutable mappings <collections.abc.MutableMapping>` of stats:-.. versionadded:: VERSION+.. versionadded:: 2.20.0.. code-block:: python@@ -342,7 +342,7 @@ one per actual value of the placeholder.:ref:`commands <topics-commands>` such as :command:`crawl` use it todecide their process exit code.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Set by :class:`~scrapy.extensions.corestats.CoreStats`.@@ -454,7 +454,7 @@ one per actual value of the placeholder.Number of requests that waited for a concurrent request for the sameresource to finish downloading.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Set by :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`.diff --git a/scrapy/commands/__init__.py b/scrapy/commands/__init__.pyindex 9eb081ada..cdc42956e 100644--- a/scrapy/commands/__init__.py+++ b/scrapy/commands/__init__.py@@ -166,7 +166,7 @@ class ScrapyCommand(ABC):being typed, where *args* are the positional arguments typed beforeit.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0See :ref:`completion`."""@@ -177,7 +177,7 @@ class ScrapyCommand(ABC):being typed, identified by *dest*, i.e. the name under which its valueis available in the ``opts`` parameter of ``run()``.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0See :ref:`completion`."""diff --git a/scrapy/core/engine.py b/scrapy/core/engine.pyindex f151cbc1f..99d1df03c 100644--- a/scrapy/core/engine.py+++ b/scrapy/core/engine.py@@ -84,7 +84,7 @@ def _supports_skip_dupefilter_once(scheduler: BaseScheduler) -> bool:class EngineState(Enum):"""The lifecycle state of :class:`ExecutionEngine`.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0The engine goes through these states in order, possibly skipping some."""@@ -274,7 +274,7 @@ class ExecutionEngine:def state(self) -> EngineState:"""The current lifecycle state of the engine.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""return self._state@@ -282,7 +282,7 @@ class ExecutionEngine:def response_active_size(self) -> int:"""Size, in bytes, that counts toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""return self.downloader.middleware._total_active_size@@ -290,7 +290,7 @@ class ExecutionEngine:def running(self) -> bool:"""Whether the engine is in the :attr:`EngineState.RUNNING` state.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Became a read-only property."""return self._state is EngineState.RUNNING@@ -331,14 +331,15 @@ class ExecutionEngine:.. versionadded:: 2.14+ .. versionchanged:: 2.20.0+ In earlier versions, if the spider was not open, it sent the+ :signal:`engine_started` signal instead of raising+ :exc:`RuntimeError`.+Completes when the engine is stopped.Raises :exc:`RuntimeError` if the spider is not open, or if the enginehas already been started or stopped.-- .. versionchanged:: VERSION- Now raises :exc:`RuntimeError` instead of returning if the spider- is not open."""if self._state is not EngineState.SPIDER_OPEN:raise RuntimeError(@@ -384,7 +385,7 @@ class ExecutionEngine:.. versionadded:: 2.14- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This is now a shortcut for :meth:`close_spider_async`, see its docsfor the behavior in different engine states, including when itcompletes.@@ -752,16 +753,18 @@ class ExecutionEngine:async def open_spider_async(self, *, close_if_idle: bool = True) -> None:"""Open the spider.+ .. versionadded:: 2.14++ .. versionchanged:: 2.20.0+ In earlier versions, a :exc:`~scrapy.exceptions.CloseSpider`+ exception raised while the spider was opening was re-raised.+Raises :exc:`RuntimeError` if a spider has already been opened.If an exception is raised during opening, the spider is stillconsidered open (the engine always transitions to the:attr:`EngineState.SPIDER_OPEN` state).- .. versionchanged:: VERSION- A :exc:`~scrapy.exceptions.CloseSpider` exception raised while the- spider is opening now closes the spider instead of being re-raised.-If the engine is requested to close the spider during opening, that isdone right after the spider is opened."""@@ -935,7 +938,7 @@ class ExecutionEngine:``mode="fast"`` was passed, it will also drop the requests left in thedownloader.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Previously, calling it while the spider was closing waited for thein-flight requests to finish, and calling it after the spider hadbeen closed raised :exc:`RuntimeError`.diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.pyindex 9ed5631ea..9bb51b97c 100644--- a/scrapy/core/scheduler.py+++ b/scrapy/core/scheduler.py@@ -68,7 +68,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):It is not inherited by subclasses that override :meth:`enqueue_request`,which must set it again.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""@classmethod@@ -108,7 +108,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):"""Process a request received by the engine.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Must handle the :reqmeta:`skip_dupefilter_once` meta key.Return ``True`` if the request is stored correctly, ``False`` otherwise.diff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex 99cc524fb..ffce8c9d1 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -395,7 +395,7 @@ class Crawler:.. versionadded:: 2.14- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Calling it while the spider is being opened now closes the spideras soon as it is open.diff --git a/scrapy/downloadermiddlewares/jsonvalidation.py b/scrapy/downloadermiddlewares/jsonvalidation.pyindex cd32114ec..5de8900b1 100644--- a/scrapy/downloadermiddlewares/jsonvalidation.py+++ b/scrapy/downloadermiddlewares/jsonvalidation.py@@ -15,7 +15,7 @@ class JsonValidationMiddleware:"""Raise :exc:`json.JSONDecodeError` for a successful:class:`~scrapy.http.JsonResponse` whose body is not valid JSON.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0.. setting:: JSONVALIDATION_ENABLEDdiff --git a/scrapy/dupefilters.py b/scrapy/dupefilters.pyindex 847ea77c5..ea5b590a0 100644--- a/scrapy/dupefilters.py+++ b/scrapy/dupefilters.py@@ -182,7 +182,7 @@ class DiskDupeFilter(RFPDupeFilter):in memory, trading some speed for a much lower memory usage on crawls thatsend a large number of requests.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Job directory contents======================diff --git a/scrapy/exceptions.py b/scrapy/exceptions.pyindex aa6203f7a..691e12563 100644--- a/scrapy/exceptions.py+++ b/scrapy/exceptions.py@@ -63,10 +63,10 @@ class CloseSpider(Exception):.. versionchanged:: 2.18.0Added support for raising it while the spider is starting.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0Added support for raising it from a downloader middleware.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0The *error* parameter.*reason* is a string with the reason for closing. *error*, keyword-only,@@ -148,7 +148,7 @@ class DecompressionError(Exception)::class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`when a response body cannot be decompressed.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""diff --git a/scrapy/exporters.py b/scrapy/exporters.pyindex 61f0f61c7..f70903158 100644--- a/scrapy/exporters.py+++ b/scrapy/exporters.py@@ -67,7 +67,7 @@ class BaseItemExporter(ABC):"""Exports the given item. This method must be implemented insubclasses.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This method may now be a coroutine function (``async def``). Scrapyawaits its result when the exporter is used by the :ref:`feedexports <topics-feed-exports>`.@@ -108,7 +108,7 @@ class BaseItemExporter(ABC)::class:`XmlItemExporter`). You must call this method before exportingany items.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This method may now be a coroutine function (``async def``). Scrapyawaits its result when the exporter is used by the :ref:`feedexports <topics-feed-exports>`.@@ -120,7 +120,7 @@ class BaseItemExporter(ABC)::class:`XmlItemExporter`). You must always call this method after youhave no more items to export.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This method may now be a coroutine function (``async def``). Scrapyawaits its result when the exporter is used by the :ref:`feedexports <topics-feed-exports>`.diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.pyindex 6f515dbe1..a33cf30d8 100644--- a/scrapy/extensions/feedexport.py+++ b/scrapy/extensions/feedexport.py@@ -181,7 +181,7 @@ class FeedStorageProtocol(Protocol):"""Open the storage for *spider* and return the file-like object thatitem exporters write into.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This method may now be a coroutine function (``async def``)."""@@ -191,7 +191,7 @@ class FeedStorageProtocol(Protocol):"""Store *file*, the file-like object returned by :meth:`open`, andclose it.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0This method may now be a coroutine function (``async def``)."""diff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.pyindex c696f8f41..115b152aa 100644--- a/scrapy/http/request/__init__.py+++ b/scrapy/http/request/__init__.py@@ -452,7 +452,7 @@ class Request(object_ref):def __await__(self) -> Generator[Any, None, Response]:"""Send this request and return its response.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0See :ref:`inline-requests`."""diff --git a/scrapy/http/response/__init__.py b/scrapy/http/response/__init__.pyindex d5d20e15f..f1d2df652 100644--- a/scrapy/http/response/__init__.py+++ b/scrapy/http/response/__init__.py@@ -311,7 +311,7 @@ class Response(object_ref):"""Return a :class:`~.Request` instance to follow the ``Location``header of this response.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0It accepts the same arguments as :meth:`follow`, except ``url``, whichcomes from the ``Location`` header. Unlike in :meth:`follow`, that URL isdiff --git a/scrapy/http/response/body_file.py b/scrapy/http/response/body_file.pyindex 0001c7bf0..3630170a9 100644--- a/scrapy/http/response/body_file.py+++ b/scrapy/http/response/body_file.py@@ -12,7 +12,7 @@ class BodyFileResponse(Response):"""A :class:`~scrapy.http.Response` whose body was written to a file, asrequested with the :reqmeta:`body_file` request meta key.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Reading :attr:`~scrapy.http.Response.body` raises :exc:`AttributeError`."""diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.pyindex 60c4c52be..6f35b8b1d 100644--- a/scrapy/http/response/text.py+++ b/scrapy/http/response/text.py@@ -52,7 +52,7 @@ class MetaRefresh(NamedTuple):"""Meta refresh of a :class:`TextResponse`, as:attr:`TextResponse.meta_refresh` reports it.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""delay: float@@ -177,7 +177,7 @@ class TextResponse(Response):def base_url(self) -> str:"""Absolute URL that relative URLs of the response are relative to.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0It is the ``href`` of the first ``<base>`` element that has one,resolved against :attr:`~Response.url`, or :attr:`~Response.url`@@ -198,7 +198,7 @@ class TextResponse(Response):""":class:`MetaRefresh` that a web browser with JavaScript enabledwould follow, or ``None``.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Setting it overrides the value read from the body."""diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.pyindex 7bd761ff1..3182b3f57 100644--- a/scrapy/linkextractors/lxmlhtml.py+++ b/scrapy/linkextractors/lxmlhtml.py@@ -219,7 +219,7 @@ class LxmlLinkExtractor::data:`scrapy.linkextractors.SUPPORTED_SCHEMES`. Use ``()`` to extractlinks regardless of their scheme.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0:type schemes: str or list:param restrict_xpaths: an XPath (or list of XPaths) that defines regionsdiff --git a/scrapy/robotstxt.py b/scrapy/robotstxt.pyindex 9313a59e3..df5e9397e 100644--- a/scrapy/robotstxt.py+++ b/scrapy/robotstxt.py@@ -78,7 +78,7 @@ class RobotParser(metaclass=ABCMeta):of the robots.txt file, or an empty iterable if there are none or thebackend does not support reading them.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0"""return ()diff --git a/scrapy/spiders/crawl.py b/scrapy/spiders/crawl.pyindex eff2bdf59..8b1b7b615 100644--- a/scrapy/spiders/crawl.py+++ b/scrapy/spiders/crawl.py@@ -66,10 +66,10 @@ class Rule:responses those links produce, and whether to keep extracting links fromthem.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0The *name* and *from_rules* parameters.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0*errback* may now be a coroutine function (``async def``) or anasynchronous generator function.diff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.pyindex a54324b00..85f3fe6a7 100644--- a/scrapy/spiders/sitemap.py+++ b/scrapy/spiders/sitemap.py@@ -237,7 +237,7 @@ class SitemapSpider(Spider):"""Return the request for the *loc* URL of the sitemap *entry*, to beparsed with *callback*.- .. versionadded:: VERSION+ .. versionadded:: 2.20.0Override this method to build those requests differently, e.g. tocarry sitemap data and request metadata over to *callback*:diff --git a/scrapy/utils/httpobj.py b/scrapy/utils/httpobj.pyindex 4a702a7de..780ac9ecf 100644--- a/scrapy/utils/httpobj.py+++ b/scrapy/utils/httpobj.py@@ -34,7 +34,7 @@ def urlparse_cached(request_or_response: Request | Response | str) -> ParseResul:class:`~scrapy.Request` or :class:`~scrapy.http.Response` object or aURL, with :func:`urllib.parse.urlparse`.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0*request_or_response* can be a URL.The result is cached and shared by everything with the same URL: for asdiff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.pyindex d0f86de90..c9a7052be 100644--- a/scrapy/utils/reactor.py+++ b/scrapy/utils/reactor.py@@ -129,7 +129,7 @@ def install_reactor() -> None:"""Install the asyncio Twisted reactor.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0*reactor_path* is now optional and accepts ``None``.*reactor_path* can be ``None`` ordiff --git a/scrapy/utils/request.py b/scrapy/utils/request.pyindex 1e349cee6..19384b6bd 100644--- a/scrapy/utils/request.py+++ b/scrapy/utils/request.py@@ -127,7 +127,7 @@ class RequestFingerprinterProtocol(Protocol):class RequestFingerprinter:"""Default fingerprinter.- .. versionchanged:: VERSION+ .. versionchanged:: 2.20.0:reqmeta:`download_handler` is taken into account.It takes into account a canonical version