Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

189 KB
Raw Blame
.. _news:

Release notes
=============

Scrapy VERSION (unreleased)
---------------------------

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   The engine status that the :ref:`telnet console <topics-telnetconsole>`
    reports now includes:

    - ``engine.state``

    - ``len(engine.requests_in_progress)``

    - ``engine.items_in_progress``

    and no longer includes:

    - ``engine.scraper.is_idle()``

    - ``engine._slot.closing``

    - ``len(engine._slot.inprogress)``

    - ``len(engine.scraper.slot.queue)``

    - ``len(engine.scraper.slot.active)``

    - ``engine.scraper.slot.itemproc_size``

    (TBD)


Scrapy 2.20.0 (2026-11-06)
--------------------------

Highlights:

-   Web-browser-like detection of response encodings, base URLs and meta
    refresh redirects, powered by frostwork_

-   Requests can be awaited to get their response

-   Non-asyncio Twisted reactors are deprecated

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Scrapy now limits by default the memory that responses can take, fixing
    `CVE-2017-14158`_.

    In previous Scrapy versions, the only limit on the total size of responses
    in memory, ``SCRAPER_SLOT_MAX_ACTIVE_SIZE``, covered responses waiting for
    or going through spider callbacks. Responses to other requests, like those
    of :ref:`media pipelines <topics-media-pipeline>`, had no limit. For
    example, when media pipelines downloaded files faster than they could store
    them, e.g. to Amazon S3, the downloaded files piled up in memory, so a
    crawled website could make Scrapy run out of memory.

    The new :setting:`RESPONSE_MAX_ACTIVE_SIZE` setting replaces
    ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` and covers all responses in memory,
    including those to :ref:`awaited requests <inline-requests>` and to
    requests sent with :meth:`ExecutionEngine.download_async()
    <scrapy.core.engine.ExecutionEngine.download_async>`. Use it to change or
    disable the limit. The :command:`shell` disables it.

    .. _CVE-2017-14158: https://github.com/advisories/GHSA-h7wm-ph43-c39p

    (:gh:`482`, :gh:`4583`, :gh:`6430`, :gh:`8417`, :gh:`8459`, :gh:`8460`)

-   :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
    now decodes at most 1 content coding per response by default, and drops
    responses that apply more with :exc:`~scrapy.exceptions.IgnoreRequest`.

    In previous Scrapy versions, the number of content codings was unlimited,
    so a malicious server could send a response with a ``Content-Encoding``
    header that chains thousands of codings (e.g. ``gzip, gzip, gzip, …``) to
    make Scrapy run as many decompression passes, causing high CPU and memory
    usage. See the `j72p-v7rm-cwm3`_ security advisory for details.

    To allow more content codings per response, set the new
    :setting:`COMPRESSION_MAX_ENCODINGS` setting.

    .. _j72p-v7rm-cwm3: https://github.com/scrapy/scrapy/security/advisories/GHSA-j72p-v7rm-cwm3

-   When :reqmeta:`http_auth_domain` is not set, the credentials from
    :reqmeta:`http_user` and :reqmeta:`http_pass` are now only sent to the
    origin (scheme, host and port) of the request where they were first used.

    In previous Scrapy versions, they were sent to any URL, so after a
    cross-origin redirect,
    :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` added
    them again to the ``Authorization`` header that
    :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` had
    removed, sending them to the redirect target. A website could redirect
    Scrapy to a host under its control to steal those credentials. See the
    `466x-p45j-m4fx`_ security advisory for details.

    .. _466x-p45j-m4fx: https://github.com/scrapy/scrapy/security/advisories/GHSA-466x-p45j-m4fx

    (:gh:`8329`)

-   When using a proxy from the ``http_proxy`` or ``https_proxy`` environment
    variables, redirects now honor the ``no_proxy`` environment variable.

    In previous Scrapy versions, a redirect to a URL with the same scheme kept
    the proxy of the original request, even if the redirect host matched
    ``no_proxy``. A website could redirect Scrapy to an internal host, and that
    request would be sent through the proxy. For ``http://`` URLs, this exposed
    the request to the proxy, including any cookies or credentials for that
    host, and allowed the proxy to forge the response. See the
    `x3px-j3qh-fp5w`_ security advisory for details.

    .. _x3px-j3qh-fp5w: https://github.com/scrapy/scrapy/security/advisories/GHSA-x3px-j3qh-fp5w

-   :class:`~scrapy.pipelines.files.S3FilesStore` now uses HTTPS with the
    default Scrapy settings.

    In previous Scrapy versions, the default value of :setting:`AWS_USE_SSL`,
    ``None``, was incorrectly interpreted to mean "disable HTTPS", so
    :class:`~scrapy.pipelines.files.S3FilesStore` used plain HTTP when talking
    to S3 unless that setting was explicitly set to ``True`` or
    :setting:`AWS_ENDPOINT_URL` was set to a custom HTTPS URL.

    An attacker who could observe that traffic could read the contents, S3
    paths and metadata of the files uploaded by
    :class:`~scrapy.pipelines.files.FilesPipeline` and
    :class:`~scrapy.pipelines.images.ImagesPipeline`, the AWS access key ID
    (but not the secret access key) and, with temporary credentials, the
    session token (``X-Amz-Security-Token``). An attacker who could tamper with
    that traffic could make Scrapy falsely think that a file is up to date or
    was successfully stored. See the `fwm7-29m4-rrj9`_ security advisory for
    details.

    To restore the previous behavior, set :setting:`AWS_USE_SSL` to ``False``.

    .. _fwm7-29m4-rrj9: https://github.com/scrapy/scrapy/security/advisories/GHSA-fwm7-29m4-rrj9

-   :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` and
    :class:`~scrapy.spiders.SitemapSpider` now only follow ``http://`` and
    ``https://`` URLs by default, plus URLs that use the scheme of the response
    they come from.

    In previous Scrapy versions, link extractors also followed ``file://`` and
    ``ftp://`` URLs, and :class:`~scrapy.spiders.SitemapSpider` followed URLs
    of any scheme. A crawled website could use that to make Scrapy read local
    files, send the :setting:`FTP_USER` and :setting:`FTP_PASSWORD` credentials
    to an FTP server of its choosing, or send requests signed with the
    :setting:`AWS_ACCESS_KEY_ID` credentials to an S3 bucket of its choosing.
    See the `q7wc-mr9r-gcjh`_ security advisory for details.

    To follow URLs with other schemes, use the new ``schemes`` parameter of
    :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` or the new
    :attr:`~scrapy.spiders.SitemapSpider.sitemap_schemes` attribute of
    :class:`~scrapy.spiders.SitemapSpider`.

    .. _q7wc-mr9r-gcjh: https://github.com/scrapy/scrapy/security/advisories/GHSA-q7wc-mr9r-gcjh

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   Dropped support for Python 3.10.
    (:gh:`8261`)

-   Dropped support for PyPy.
    (:gh:`7849`, :gh:`8358`, :gh:`8367`, :gh:`8393`)

-   New dependencies:

    - aiodns_ >= 3.3.0

    - aioftp_ >= 0.26.1

    - formerly_ >= 0.1.0

    - frostwork_ >= 0.2.2

    - typing-extensions_ >= 4.13.0 (on Python 3.12 and lower)

    (:gh:`7894`, :gh:`8213`, :gh:`8303`, :gh:`8376`, :gh:`8378`, :gh:`8393`)

-   charset-normalizer_ is no longer a dependency.
    (:gh:`8393`)

-   Increased the minimum versions of the following dependencies:

    - cssselect_: 0.9.1 → 1.2.0

    - itemloaders_: 1.0.1 → 1.0.5

    - lxml_: 4.6.4 → 4.9.4

    - parsel_: 1.5.0 → 1.8.1

    - Protego_: 0.1.15 → 0.2.0

    - queuelib_: 1.6.1 → 1.8.0

    - Twisted_: 21.7.0 → 23.10.0

    - w3lib_: 2.1.1 → 2.6.0

    (:gh:`6430`, :gh:`8195`, :gh:`8261`, :gh:`8303`, :gh:`8393`, :gh:`8460`)

-   Changes to :ref:`extras <extras>`:

    - New ``clipboard``, ``memusage`` and ``socks`` extras.

    - ``gcs``: google-cloud-storage_ 1.29.0 → 1.30.0

    - ``httpx``: httpx2_ 2.0.0 → 2.3.0

    - ``images``: Pillow_ 8.3.2 → 9.2.0

    - ``s3``: boto3_ 1.20.0 → 1.35.2, and aioboto3_ >= 9.0.0 is now
      included.

    - ``twisted-http2``: Twisted_ 21.7.0 → 23.10.0

    - ``uvloop``: uvloop_ 0.16.0 → 0.17.0

    (:gh:`6430`, :gh:`7826`, :gh:`7932`, :gh:`8058`, :gh:`8062`, :gh:`8162`,
    :gh:`8213`, :gh:`8261`, :gh:`8330`, :gh:`8341`, :gh:`8460`)

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   Changes to the ``fetch()`` shortcut of the :ref:`Scrapy shell
    <topics-shell>`:

    - Requests are now downloaded with
      :meth:`ExecutionEngine.download_async()
      <scrapy.core.engine.ExecutionEngine.download_async>` instead of being
      passed to the scheduler. As a result, spider middlewares and duplicate
      filtering are no longer applied to them.

    - When a :class:`~scrapy.Request` object is passed, its ``callback`` and
      ``errback`` are no longer called, and a warning is issued if they are
      set.

    - The engine is no longer started, so the :signal:`engine_started` and
      :signal:`engine_stopped` signals are not sent, and components that start
      on them, such as the :class:`~scrapy.extensions.memusage.MemoryUsage`
      extension and the :ref:`telnet console <topics-telnetconsole>`, do not
      run in the shell.

    (:gh:`8174`, :gh:`8176`, :gh:`8179`)

-   Changes to the behavior of :class:`~scrapy.core.engine.ExecutionEngine`
    methods that change its state:

    - :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` now also
      stops the engine. Calling it while the spider is opening now closes the
      spider as soon as it is open, calling it while the spider is closing now
      completes immediately instead of waiting for the in-flight requests, and
      calling it after the spider has been closed no longer raises
      :exc:`RuntimeError`.

    - :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` now just calls
      :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` with the
      ``shutdown`` reason. Calling it after
      :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` and before
      :meth:`~scrapy.core.engine.ExecutionEngine.start_async` now closes the
      spider instead of raising :exc:`RuntimeError`. Calling it while the
      spider is closing or after the engine has stopped now completes
      immediately, instead of raising :exc:`RuntimeError` or waiting for the
      in-flight requests.

    - :meth:`~scrapy.core.engine.ExecutionEngine.start_async` now raises
      :exc:`RuntimeError` if called before
      :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` has
      finished, instead of sending the :signal:`engine_started` signal
      anyway, and if called after the engine has stopped.

    - The :signal:`engine_stopped` signal is now sent only after all
      :signal:`engine_started` handlers have finished.

    - If the spider is closed before the engine is started, for example by
      calling :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`
      from a :signal:`spider_opened` handler, the :signal:`engine_started` and
      :signal:`engine_stopped` signals are not sent.

    - :meth:`Crawler.stop_async() <scrapy.crawler.Crawler.stop_async>`, when
      called before the engine is running, e.g. while the spider is being
      opened, now closes the spider as soon as it is open, instead of being
      ignored.

    - A :exc:`~scrapy.exceptions.CloseSpider` exception raised while the spider
      is opening now closes the spider as soon as it is open, instead of being
      re-raised by
      :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`.

    - ``crawl()`` and
      :meth:`~scrapy.core.engine.ExecutionEngine.download_async` now raise
      :exc:`RuntimeError` when called while the spider is closing, once the
      close has stopped waiting for the requests in progress.

    (:gh:`6916`, :gh:`8217`, :gh:`8272`, :gh:`8319`)

-   Setting the :attr:`ExecutionEngine.running
    <scrapy.core.engine.ExecutionEngine.running>` attribute no longer has any
    effect, and is deprecated. Its value is now ``False`` while the spider is
    closing.
    (:gh:`8181`)

-   :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now
    enforces :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE`, also
    when using :reqmeta:`ftp_local_filename`, so downloads bigger than 1 GiB
    now fail by default. To download bigger files, raise the limit to a size
    that fits them, with :setting:`DOWNLOAD_MAXSIZE` or with the
    :reqmeta:`download_maxsize` request meta key for specific requests. You can
    also set the limit to ``0`` to lift it altogether.
    (:gh:`8213`)

-   The built-in HTTP download handlers now limit their open connections with
    the new :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER` setting, whose
    default value is derived from the limit of open files of the process, or
    is 480 on Windows.
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` had no
    such limit before, and
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    and :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`
    used :setting:`CONCURRENT_REQUESTS` as their limit. If you set
    :setting:`CONCURRENT_REQUESTS` higher than that default, a warning is
    logged, and you need to set :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER`
    or, on platforms other than Windows, raise the limit of open files of the
    process (see :ref:`faq-too-many-open-files`).
    (:gh:`2653`, :gh:`5024`, :gh:`5618`, :gh:`7932`)

-   :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no
    longer falls back to the ``httpx`` package when httpx2_ is not installed.
    Install the ``httpx`` :ref:`extra <extras>` to use it.
    (:gh:`7932`)

-   Responses that your code keeps in memory, e.g. in the
    :attr:`.Request.meta` of scheduled requests or in component attributes,
    now count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE` until they are
    released, and so do responses in a reference cycle until Python's garbage
    collector frees them. Keeping many responses can now slow down your crawl.
    To find out whether your crawl keeps responses alive, see
    :ref:`topics-leaks`.

    If requests have been held back by the limit for 5 seconds with none being
    downloaded and no response released, Scrapy runs Python's garbage collector
    and, if that releases no response, lets one request through, logging a
    warning the first time and increasing the new
    :stat:`response_max_active_size/bypass_count` stat.

    Scrapy itself no longer keeps responses alive in reference cycles after a
    request fails, e.g. with an
    :exc:`~scrapy.spidermiddlewares.httperror.HttpError` that no errback
    handles, or after a failed media pipeline download.

    (:gh:`6430`, :gh:`8417`, :gh:`8459`, :gh:`8460`)

-   Header names in ``Headers`` objects, i.e. in :attr:`Request.headers
    <scrapy.Request.headers>` and :attr:`Response.headers
    <scrapy.http.Response.headers>`, now keep the case they are first set
    with, instead of being converted to ``Title-Case``.
    Lookups are still case-insensitive, but code that iterates over header
    names and compares them to ``Title-Case`` strings must be updated.

    Over HTTP/1.1,
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    and :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`
    now send header names in the case you write them in, and keep the case of
    response header names as received. See the :ref:`comparison of download
    handlers <download-handlers-ref>`.

    (:gh:`2711`, :gh:`7893`, :gh:`8265`)

-   :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`, and hence
    :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`, now sets the
    new :reqmeta:`skip_dupefilter_once` request meta key on retry requests
    instead of setting their :attr:`~scrapy.Request.dont_filter` attribute to
    ``True``. As a result, with schedulers that support that meta key, like
    the default one, requests derived from a retry request, e.g. redirects,
    are no longer exempt from duplicate filtering.
    (:gh:`6895`, :gh:`8224`)

-   Requests whose meta is copied from a downloaded request, e.g. redirects and
    retries, are now assigned to the download slot of their own URL domain,
    instead of keeping the slot of the original request. Requests whose meta is
    copied from a request whose :reqmeta:`download_slot` request meta key you
    set keep that slot, so set it to keep redirects and retries in the slot of
    the original request.
    (:gh:`8337`)

-   :reqmeta:`http_user` and :reqmeta:`http_pass` are now only sent to the
    origin (scheme, host and port) of the request that you set them on, so
    they are no longer sent after a cross-origin redirect. To send them to a
    different origin, set the :reqmeta:`http_auth_domain` request meta key.
    :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` also no
    longer sets the ``Authorization`` header of requests whose URL scheme is
    neither ``http`` nor ``https``.
    (:gh:`8329`)

-   Keys of :ref:`component priority dictionaries
    <component-priority-dictionaries>`, such as
    :setting:`DOWNLOADER_MIDDLEWARES`, that cannot be resolved into a
    component, e.g. the import path of a component that no longer exists, now
    raise :exc:`ValueError`, even if their priority is ``None``. Some of them
    used to be silently ignored, and a missing module used to raise
    :exc:`ModuleNotFoundError` instead. Fix the import path or remove the
    entry.
    (:gh:`7820`, :gh:`8159`)

-   :class:`HttpCompressionMiddleware
    <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`
    now raises :exc:`~scrapy.exceptions.DecompressionError` for a response body
    that cannot be decompressed, instead of the exception of the underlying
    decompression library, e.g. :exc:`zlib.error`, :exc:`OSError` or
    :exc:`EOFError`. Update any code that catches those exceptions, e.g. in
    request errbacks.
    (:gh:`8177`)

-   ``str()`` of a :class:`~scrapy.Spider` object now returns its
    :attr:`~scrapy.Spider.name`. Use ``repr()`` to get the previous output,
    e.g. ``<MySpider 'my_spider' at 0x7f0e2c1a3b50>``.
    (:gh:`1576`, :gh:`8086`)

-   The ``scrapy.utils.defer.parallel()`` and
    ``scrapy.utils.defer.parallel_async()`` functions, now deprecated, return a
    :class:`~twisted.internet.defer.Deferred` that fires with ``None``, instead
    of with the result list of a :class:`~twisted.internet.defer.DeferredList`.
    (:gh:`5182`, :gh:`7959`)

-   The ``process_response()`` method of :class:`HttpCompressionMiddleware
    <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`
    is now a coroutine function. Code that calls it must now await it, and
    subclasses that override it and call ``super().process_response()`` must
    now be ``async def``.
    (:gh:`3751`, :gh:`8153`)

-   The ``item_scraped()`` method of
    ``scrapy.extensions.feedexport.FeedExporter``, and the
    ``start_exporting()`` and ``finish_exporting()`` methods of
    ``scrapy.extensions.feedexport.FeedSlot``, are now coroutine functions.
    (:gh:`5622`, :gh:`6273`, :gh:`7800`)

-   ``parent_id`` is now part of :attr:`Request.attributes
    <scrapy.Request.attributes>`, so :meth:`Request.replace()
    <scrapy.Request.replace>` and
    :func:`~scrapy.utils.request.request_from_dict` pass a ``parent_id``
    keyword argument to the request class. :class:`~scrapy.Request` subclasses
    whose ``__init__()`` method accepts neither ``parent_id`` nor
    ``**kwargs`` must be updated.
    (:gh:`8225`)

-   :ref:`Item exporters <topics-exporters>` now turn the values of fields
    without a serializer into lists if they are list-like, e.g. tuples or sets,
    and into dictionaries if they are items, recursively. This changes the
    output of :class:`~scrapy.exporters.PickleItemExporter`,
    :class:`~scrapy.exporters.MarshalItemExporter`,
    :class:`~scrapy.exporters.PprintItemExporter` and custom
    :class:`~scrapy.exporters.BaseItemExporter` subclasses, and
    :class:`~scrapy.exporters.CsvItemExporter` now joins list-like values,
    e.g. sets, the way it joins lists.
    (:gh:`3888`, :gh:`3890`, :gh:`8057`)

-   The output of a request :ref:`errback <errbacks>` that runs because a
    download failed now goes through the
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
    methods of :ref:`spider middlewares <topics-spider-middleware>`, with
    ``None`` as the response, and exceptions raised while iterating that output
    reach their
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
    methods. Spider middlewares whose methods expect a response must now also
    handle ``None``. Built-in spider middlewares now also process that output,
    e.g. :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` can
    drop requests from such errbacks, and
    :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware` now sets
    :reqmeta:`is_start_request` in its ``process_start()`` method instead of in
    ``get_processed_request()``.
    (:gh:`4999`, :gh:`7992`)

-   Feed storages now refuse :ref:`feed modes <feed-mode>` that they do not
    support, which prevents the crawl from starting. In particular, setting
    the ``overwrite`` feed option to ``False`` for an :ref:`S3
    <topics-feed-storage-s3>` feed, which used to log a warning, now raises
    :exc:`ValueError`. For a :ref:`GCS <topics-feed-storage-gcs>` feed, which
    used to log a warning and overwrite the file, it now appends to the file.
    (:gh:`5580`, :gh:`7826`, :gh:`8173`)

-   :attr:`~scrapy.spiders.SitemapSpider.sitemap_follow` now also filters the
    sitemap URLs declared in :file:`robots.txt` files. If you set it, make sure
    it matches the URLs of the sitemaps declared in the :file:`robots.txt`
    files that you want to follow.
    (:gh:`1182`, :gh:`8175`)

-   The ``process_request()`` method of
    :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` is now
    a coroutine function. Code that calls it must now await it, and subclasses
    that override it and call ``super().process_request()`` must now be
    ``async def``.
    (:gh:`2744`, :gh:`7915`)

-   ``XMLFeedSpider.parse_nodes()`` and ``CSVFeedSpider.parse_rows()`` are now
    asynchronous generators. Code that calls them must now iterate over their
    output with ``async for``.
    (:gh:`8392`)

-   The settings of the built-in storages of :ref:`media pipelines
    <topics-media-pipeline>` are no longer set as class attributes of those
    storages, e.g. ``S3FilesStore.POLICY`` keeps its default value regardless
    of :setting:`FILES_STORE_S3_ACL`. Read them from the storage object of the
    pipeline instead, e.g. ``pipeline.store.POLICY``.
    (:gh:`7925`)

-   ``Scraper.slot`` and the deprecated ``scrapy.core.scraper.Slot`` class
    now take a :class:`~scrapy.crawler.Crawler` object instead of a maximum
    active size, their ``finish_response()`` method no longer takes a
    *result* parameter, their ``MIN_RESPONSE_SIZE`` attribute was removed,
    and their deprecated ``active_size`` attribute is read-only.
    (:gh:`6430`)

-   The engine status that the :ref:`telnet console <topics-telnetconsole>`
    reports now includes ``engine.response_active_size`` instead of
    ``engine.scraper.slot.active_size`` and
    ``engine.scraper.slot.needs_backout()``.
    (:gh:`6430`)

-   The deprecated ``METAREFRESH_IGNORE_TAGS`` setting and the *ignore_tags*
    parameter of the deprecated ``scrapy.utils.response.get_meta_refresh()``
    function now raise :exc:`ValueError` for tags other than ``script``,
    ``noscript`` and ``template``, and only whether ``noscript`` is listed has
    any effect. ``METAREFRESH_IGNORE_TAGS`` is ignored altogether if
    :setting:`METAREFRESH_SCRIPTING` is set with the same or a higher
    :ref:`priority <populating-settings>`.
    (:gh:`8393`)

-   Scrapy no longer calls the deprecated ``spider_opened()`` and
    ``spider_closed()`` methods of
    :class:`~scrapy.extensions.spiderstate.SpiderState`, so overriding them in
    a subclass has no effect. Handle the :signal:`spider_state_loaded` and
    :signal:`spider_state_saving` signals instead.
    (:gh:`5153`, :gh:`7650`, :gh:`7656`)

-   Due to the new minimum version of w3lib_, 2.6.0:

    - Request URLs are now normalized the way web browsers normalize them, so
      :attr:`Request.url <scrapy.Request.url>` and :attr:`Response.url
      <scrapy.http.Response.url>` can differ from the URL you pass, e.g.
      ``https://EXAMPLE.com`` becomes ``https://example.com/``,
      ``https://example.com/a/../b`` becomes ``https://example.com/b``, and
      ``https://0x7f.1/`` becomes ``https://127.0.0.1/``. Update any code that
      compares request or response URLs to literal strings.

    - :ref:`Request fingerprints <request-fingerprints>` might differ for some
      URLs, so that cached responses of
      :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` and
      the seen requests of a :ref:`job directory <job-dir>` from an earlier
      version might not match the same requests.

    (:gh:`8393`)

-   For encodings declared in the response, :attr:`TextResponse.encoding
    <scrapy.http.TextResponse.encoding>` now reports the Python codec names of
    the encodings that web browsers use, e.g. ``gbk`` instead of ``gb18030``
    for a ``gb2312`` label, and ``utf-16-le`` instead of ``utf-16``. It also
    reports ``utf-8`` instead of ``cp1252`` when the response declares no
    encoding and its body is ASCII-only, e.g. empty. For encoding labels that
    web browsers decode as a single U+FFFD character, like ``hz``, it reports
    ``replacement``, which raises :exc:`UnicodeDecodeError` when decoding any
    non-empty body, so use :attr:`TextResponse.text
    <scrapy.http.TextResponse.text>` instead of
    ``response.body.decode(response.encoding)``.

    :attr:`TextResponse.text <scrapy.http.TextResponse.text>` can also differ:

    - The detected encoding of responses that declare none and are not valid
      UTF-8 can change.

    - Encodings that web browsers ignore, like ``utf-32`` or ``cp437``, are now
      ignored when declared, and the encoding is detected instead. A ``<meta>``
      tag or XML declaration that declares UTF-16 now means UTF-8.

    - Encoding declarations in ``<meta>`` tags are now found past the first
      4096 bytes of the response body.

    - Response bodies are decoded the way web browsers decode them, which for
      some encodings, like ``euc-jp``, differs from Python codecs.

    (:gh:`8393`)

-   The base URL of a response, used by :meth:`TextResponse.urljoin()
    <scrapy.http.TextResponse.urljoin>`, :ref:`selectors <topics-selectors>`
    and :ref:`link extractors <topics-link-extractors>`, and the meta refresh
    that :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware`
    follows, are now found closer to the way web browsers find them. In
    particular, they are now found anywhere in the response body, instead of
    only in its first 4096 characters, the meta refresh with the shortest
    delay, and the last one among those with that delay, now wins over the
    first one, meta refreshes within ``<script>`` or ``<template>`` elements,
    or to the response URL itself, ignoring its fragment, are no longer
    followed, and the base URL is normalized like request URLs.
    (:gh:`8393`)

-   The ``response`` attribute of :class:`~scrapy.Selector` objects is now
    read-only. For the selector of a response, i.e.
    :attr:`TextResponse.selector <scrapy.http.TextResponse.selector>`, it no
    longer keeps the response alive, and becomes ``None`` once nothing else
    refers to the response.
    (:gh:`8245`)

-   A negative value of :setting:`FILES_EXPIRES` or :setting:`IMAGES_EXPIRES`
    now disables expiration, so that files are never downloaded again once
    fetched. Previously, it made files be downloaded again every time; set
    the setting to ``0`` for that.
    (:gh:`8046`)

-   Crawls closed because :meth:`~scrapy.Spider.start` raised an exception, or
    by :setting:`CLOSESPIDER_ERRORCOUNT`, now make the :command:`crawl` and
    :command:`runspider` commands exit with code 1.
    (:gh:`8082`, :gh:`8203`)

-   A blank line between :ref:`contract <topics-contracts>` lines in a callback
    docstring now starts a new batch of contracts, which needs its own ``@url``
    line; otherwise, the contracts of that batch do not run.

    ``ContractsManager.extract_contracts()`` now returns a list of batches,
    each a list of contracts, and ``ContractsManager.from_method()`` now
    returns a list of requests instead of a request or ``None``.

    (:gh:`768`, :gh:`8043`)

Deprecations
~~~~~~~~~~~~

-   The following modules are deprecated, with no replacement:

    - ``scrapy.utils.benchserver``

    - ``scrapy.utils.console``

    - ``scrapy.utils.datatypes``

    - ``scrapy.utils.display``

    - ``scrapy.utils.ftp``

    - ``scrapy.utils.gz``

    - ``scrapy.utils.job``

    - ``scrapy.utils.ossignal``

    - ``scrapy.utils.sitemap``

    - ``scrapy.utils.ssl``

    - ``scrapy.utils.template``

    - ``scrapy.utils.versions``

    (:gh:`7766`, :gh:`8135`, :gh:`8146`, :gh:`8270`)

-   The following functions, classes and methods are deprecated, with no
    replacement:

    - ``scrapy.core.scraper.Scraper.handle_spider_error()``

    - ``scrapy.spiders.sitemap.iterloc()``

    - ``scrapy.utils.defer.aiter_errback()``

    - ``scrapy.utils.defer.iter_errback()``

    - ``scrapy.utils.defer.parallel()``

    - ``scrapy.utils.defer.parallel_async()``

    - ``scrapy.utils.python.MutableAsyncChain``

    (:gh:`8175`, :gh:`8288`, :gh:`8350`, :gh:`8363`, :gh:`8414`)

-   ``scrapy.utils.python.garbage_collect()`` is deprecated, use
    :func:`gc.collect` instead.
    (:gh:`8367`)

-   Passing a coroutine to ``scrapy.utils.spider.iterate_spider_output()``, for
    which it returns a :class:`~twisted.internet.defer.Deferred` instead of an
    iterable, is deprecated. Await the coroutine and pass its result instead.
    (:gh:`8394`)

-   Passing a synchronous iterable to
    ``scrapy.core.scraper.Scraper.handle_spider_output_async()`` is deprecated,
    pass an asynchronous iterable instead.
    (:gh:`8288`)

-   ``scrapy.utils.reactor.is_asyncio_reactor_installed()`` is deprecated. To
    check for asyncio support, use
    :func:`~scrapy.utils.asyncio.is_asyncio_available`. To check whether a
    reactor is installed, use ``scrapy.utils.reactor.is_reactor_installed()``.
    See :ref:`asyncio-old-versions` for code that must also support older
    Scrapy versions.
    (:gh:`8128`, :gh:`8454`)

-   Defining a ``parse_item()`` method in a
    :class:`~scrapy.spiders.XMLFeedSpider` subclass is deprecated. Define
    :meth:`~scrapy.spiders.XMLFeedSpider.parse_node` instead.
    (:gh:`8147`)

-   Exceptions that the ``process_response()`` method of a downloader
    middleware raises can now reach the ``process_exception()`` methods of the
    downloader middlewares that have not processed the response yet, so that
    :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` can retry
    them. Because existing ``process_exception()`` methods may not expect those
    exceptions, e.g. the :exc:`~scrapy.exceptions.IgnoreRequest` raised to drop
    a response, this only happens when the new
    :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` setting is ``True``.
    Its default value, ``False``, is deprecated, and a warning is issued when
    an exception would reach the ``process_exception()`` method of a
    non-Scrapy downloader middleware if the setting were ``True``. Check that
    those methods handle such exceptions as intended, and set the setting to
    ``True``. New projects generated with :command:`startproject` already do.
    (:gh:`7790`, :gh:`8177`)

-   ``scrapy.utils.deprecate.create_deprecated_class()`` is deprecated, use
    ``formerly.deprecated_class()`` from formerly_ instead.
    ``scrapy.utils.decorators.deprecated()`` is deprecated, use
    :func:`warnings.deprecated` instead (or ``typing_extensions.deprecated()``
    on Python 3.12 and lower).
    (:gh:`7894`)

-   ``scrapy.utils.url.add_http_if_no_scheme()`` is deprecated. Use
    :func:`w3lib.url.add_http_if_no_scheme` instead.
    (:gh:`2186`, :gh:`8189`)

-   The following methods of ``Headers`` are deprecated:

    - ``fromkeys()``: pass ``(key, value)`` pairs to ``Headers`` instead.

    - ``has_key()``: use the ``in`` operator instead.

    - ``normkey()`` and ``normvalue()``.

    - ``setlist()``: assign the list instead, e.g. ``headers[key] =
      values``.

    - ``setlistdefault()``: use ``setdefault()`` instead.

    - ``to_unicode_dict()``: use ``to_tuple_list()`` instead.

    (:gh:`8259`, :gh:`8264`)

-   The ``spider_closed_callback`` parameter of
    :class:`~scrapy.core.engine.ExecutionEngine` is deprecated. Closing the
    spider now stops the engine, and :meth:`Crawler.crawl_async()
    <scrapy.crawler.Crawler.crawl_async>` completes when that happens.
    (:gh:`6916`, :gh:`8217`)

-   Setting the :attr:`ExecutionEngine.running
    <scrapy.core.engine.ExecutionEngine.running>` attribute is deprecated, and
    has no effect.
    (:gh:`8181`)

-   Custom :ref:`schedulers <topics-scheduler>` that do not set
    :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`
    to ``True`` are deprecated, including subclasses of the default scheduler
    that override its
    :meth:`~scrapy.core.scheduler.BaseScheduler.enqueue_request` method.
    Scrapy turns :reqmeta:`skip_dupefilter_once` into ``dont_filter=True`` for
    them, with a warning. To update a scheduler,
    handle :reqmeta:`skip_dupefilter_once` in its
    :meth:`~scrapy.core.scheduler.BaseScheduler.enqueue_request` method and
    set
    :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`
    to ``True``.
    (:gh:`6895`, :gh:`8224`)

-   The ``overwrite`` feed option is deprecated. Use the new ``mode``
    :ref:`feed option <feed-mode>` instead: ``"overwrite"`` instead of
    ``True``, ``"append"`` instead of ``False``. The ``overwrite`` attribute of
    :class:`~scrapy.extensions.feedexport.FTPFeedStorage` is deprecated as
    well.

    The default value of the new :setting:`FEED_MODE` setting, ``None``, is
    also deprecated: it will change to ``"create"`` in a future version of
    Scrapy, so that existing feed targets are not modified. Set
    :setting:`FEED_MODE`, or the ``mode`` option of every feed, to silence the
    deprecation warning. New projects set :setting:`FEED_MODE` to ``"create"``
    in their :file:`settings.py` file.
    (:gh:`5580`, :gh:`7826`)

-   The ``STORE_SCHEMES`` attribute of
    :class:`~scrapy.pipelines.files.FilesPipeline` is deprecated. Use the new
    :setting:`MEDIA_STORAGES` setting instead.
    (:gh:`5991`, :gh:`7925`)

-   The ``spider_opened()`` and ``spider_closed()`` methods of
    :class:`~scrapy.extensions.spiderstate.SpiderState` are deprecated. Use the
    new :signal:`spider_state_loaded` and :signal:`spider_state_saving`
    signals instead.
    (:gh:`5153`, :gh:`7650`, :gh:`7656`)

-   The *spider* and *to_native_str_type* parameters of
    ``scrapy.robotstxt.decode_robotstxt()`` are deprecated and have no effect.
    (:gh:`8316`)

-   Calling :meth:`AsyncCrawlerRunner.crawl()
    <scrapy.crawler.AsyncCrawlerRunner.crawl>` without a running asyncio event
    loop while :setting:`TWISTED_REACTOR_ENABLED` is ``False`` is deprecated.
    Call it from a coroutine that runs in the event loop instead, e.g. one
    passed to :func:`asyncio.run`.
    (:gh:`7201`, :gh:`7681`, :gh:`8327`)

-   ``ImagesPipeline.convert_image()`` has a new *image_format* keyword-only
    parameter with the original image format. Overriding the method without
    that parameter, and calling it without it, are deprecated.
    (:gh:`8198`, :gh:`8457`)

-   The ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` setting is deprecated. Use
    :setting:`RESPONSE_MAX_ACTIVE_SIZE` instead.

    ``scrapy.core.scraper.Slot`` is deprecated as well, and so are the
    ``active_size`` and ``max_active_size`` attributes and the
    ``needs_backout()`` method of ``Scraper.slot``. Use
    :attr:`ExecutionEngine.response_active_size
    <scrapy.core.engine.ExecutionEngine.response_active_size>`,
    :setting:`RESPONSE_MAX_ACTIVE_SIZE` and
    :meth:`ExecutionEngine.needs_backout()
    <scrapy.core.engine.ExecutionEngine.needs_backout>` instead.

    (:gh:`6430`)

-   Using a Twisted reactor other than
    :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` is
    deprecated, be it set through the ``TWISTED_REACTOR`` setting, installed
    by code outside Scrapy, or passed to
    :func:`~scrapy.utils.reactor.install_reactor`. Setting ``TWISTED_REACTOR``
    to ``None`` or ``""`` is deprecated as well. Remove any custom
    ``TWISTED_REACTOR`` setting from your code.

    The ``FORCE_CRAWLER_PROCESS`` setting is deprecated as well.

    (:gh:`8111`, :gh:`8454`)

-   Passing an encoding that web browsers do not support, e.g. ``cp437``, to
    the *encoding* parameter of :class:`~scrapy.http.TextResponse` is
    deprecated. To decode a response body with such an encoding, use
    ``response.body.decode(encoding)``.
    (:gh:`8393`)

-   ``scrapy.utils.response.get_base_url()`` and
    ``scrapy.utils.response.get_meta_refresh()`` are deprecated. Use the new
    :attr:`TextResponse.base_url <scrapy.http.TextResponse.base_url>` and
    :attr:`TextResponse.meta_refresh <scrapy.http.TextResponse.meta_refresh>`
    attributes instead. When there is no meta refresh, the latter is ``None``
    instead of ``(None, None)``.
    (:gh:`8393`)

-   The ``METAREFRESH_IGNORE_TAGS`` setting is deprecated. Use the new
    :setting:`METAREFRESH_SCRIPTING` setting instead.
    (:gh:`8393`)

New features
~~~~~~~~~~~~

-   :class:`~scrapy.Request` objects can now be awaited, e.g. in a spider
    callback, to send them and get their response. See :ref:`inline-requests`.
    (:gh:`1144`, :gh:`8207`)

-   Added a :setting:`DOWNLOAD_HANDLERS_BY_NAME` setting and a
    :reqmeta:`download_handler` request meta key, to :ref:`choose a download
    handler per request <download-handler-ids>`, e.g. a browser-based handler
    for only some requests. The default :ref:`request fingerprinter
    <request-fingerprints>` takes :reqmeta:`download_handler` into account
    when it does not match the URL scheme.

    :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
    copies :reqmeta:`download_handler` from the request that triggers a
    ``robots.txt`` request, and sets the new :reqmeta:`is_robotstxt_request`
    request meta key on the ``robots.txt`` requests that it sends.

    (:gh:`5510`, :gh:`7885`)

-   Added a :reqmeta:`body_file` request meta key, which writes the response
    body into a file as it is downloaded, so that large responses can be
    downloaded without keeping them in memory, with :ref:`download handlers
    that support it <download-handlers-ref>`. Set it to a file path, or to
    ``True`` to write into a new temporary file. Those requests get a
    :class:`~scrapy.http.BodyFileResponse`, and neither they nor their
    responses count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`.
    (:gh:`3880`, :gh:`4620`, :gh:`5044`, :gh:`7987`, :gh:`8466`)

-   Sending a shutdown signal, e.g. with Ctrl+C, a second time now triggers a
    fast stop, which drops queued and in-progress downloads instead of waiting
    for them to finish. Sending it a third time forces an unclean shutdown,
    which used to happen on the second signal. Dropped requests are counted in
    the new ``downloader/request_dropped_count`` stat.

    :meth:`Crawler.stop_async() <scrapy.crawler.Crawler.stop_async>`,
    :meth:`CrawlerRunner.stop() <scrapy.crawler.CrawlerRunner.stop>` and
    :meth:`AsyncCrawlerRunner.stop() <scrapy.crawler.AsyncCrawlerRunner.stop>`
    have a new keyword-only ``mode`` parameter that can be ``"graceful"`` (the
    default), ``"fast"`` or ``"force"``, which works as ``"fast"`` unless you
    use :class:`~scrapy.crawler.CrawlerProcess` or
    :class:`~scrapy.crawler.AsyncCrawlerProcess`.
    :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` and
    :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` accept
    ``"graceful"`` and ``"fast"``. A custom :setting:`DOWNLOADER` class needs
    a ``stop()`` coroutine method, returning the number of dropped requests,
    for fast stops to drop its downloads.
    (:gh:`4749`, :gh:`5000`, :gh:`7455`, :gh:`8296`, :gh:`8312`)

-   Added the :reqmeta:`delay` request meta key, which holds a request for at
    least the given number of seconds before sending it, and the
    :reqmeta:`delayed` request meta key, which lists the delays that a request
    has been held for.
    (:gh:`802`, :gh:`3835`, :gh:`4980`, :gh:`5015`, :gh:`7873`)

-   Added :attr:`TextResponse.base_url <scrapy.http.TextResponse.base_url>` and
    :attr:`TextResponse.meta_refresh <scrapy.http.TextResponse.meta_refresh>`,
    which return the base URL of a response and the
    :class:`~scrapy.http.response.text.MetaRefresh` that a web browser with
    JavaScript enabled would follow, and which you can set to override what the
    response body says. When :setting:`METAREFRESH_SCRIPTING` is ``False``,
    :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware`
    ignores an overridden :attr:`~scrapy.http.TextResponse.meta_refresh`.
    (:gh:`8393`)

-   Added a :setting:`METAREFRESH_SCRIPTING` setting, which you can set to
    ``False`` to follow the meta refresh that a web browser with JavaScript
    disabled would follow.
    (:gh:`8393`)

-   Added a :setting:`FEED_MODE` setting and a ``mode`` :ref:`feed option
    <feed-options>`, which set what happens when the target of a feed already
    exists: ``"append"`` to it, ``"overwrite"`` it, or ``"create"``, which
    leaves it untouched, logs an error and increases the
    ``feedexport/conflicts/<storage class>`` stat. Custom feed storages declare
    the modes that they support with the new
    :attr:`~scrapy.extensions.feedexport.FeedStorageProtocol.supported_modes`
    attribute.

    The ``-o`` command-line option also follows :setting:`FEED_MODE`, so in a
    project with :setting:`FEED_MODE` set to ``"create"``, ``-o`` leaves an
    existing file untouched. Use ``-O`` to overwrite it.
    (:gh:`5580`, :gh:`7826`)

-   Added :class:`~scrapy.dupefilters.DiskDupeFilter`, a
    :setting:`DUPEFILTER_CLASS` that keeps request fingerprints in an SQLite
    database on disk instead of in memory, to lower memory usage on large
    crawls.
    (:gh:`5307`, :gh:`7952`)

-   Added a :setting:`JOBDIR_SYNC_EVERY` setting to write the
    :file:`active.json` file of the :ref:`job directory <job-dir>`, and the
    database of :class:`~scrapy.dupefilters.DiskDupeFilter`, while the crawl
    runs, so that a crawl killed before it can stop cleanly can resume from
    its latest state, provided that :setting:`SCHEDULER_DISK_QUEUE` and
    :setting:`SCHEDULER_START_DISK_QUEUE` use SQLite-based queues. Custom
    :setting:`SCHEDULER_PRIORITY_QUEUE` classes need the new
    :attr:`~scrapy.pqueues.PriorityQueueProtocol.changed` attribute and
    :meth:`~scrapy.pqueues.PriorityQueueProtocol.state` method to support it.

    :file:`active.json` is also now written atomically, so that it cannot be
    left half-written.

    (:gh:`7978`, :gh:`8152`, :gh:`8322`)

-   Added :attr:`Request.id <scrapy.Request.id>`, a unique identifier of each
    request, and :attr:`Request.parent_id <scrapy.Request.parent_id>`, the
    :attr:`~scrapy.Request.id` of the request it derives from, which Scrapy
    sets for requests from callbacks and errbacks, from
    :meth:`Response.follow() <scrapy.http.Response.follow>` and from
    :meth:`Request.replace() <scrapy.Request.replace>`, e.g. redirects and
    retries. Use the new *parent_id* parameter of :class:`~scrapy.Request` to
    set it on requests that your code builds on behalf of another request, e.g.
    in a downloader middleware. Both survive serialization, e.g. into the disk
    queues of a :ref:`job directory <job-dir>`.
    (:gh:`8225`)

-   Added a :command:`genrequest` command, which prints the Python code of a
    :class:`~scrapy.Request` object equivalent to a cURL command, given as an
    argument or read from the system clipboard with the new ``clipboard``
    :ref:`extra <extras>`.
    (:gh:`3990`, :gh:`3991`, :gh:`8058`)

-   Added a :command:`complete` command, which prints a :ref:`shell completion
    <completion>` script for Bash, fish or Zsh. Completion covers the spiders,
    settings and templates of the current project, and custom commands, which
    can provide their own candidates by overriding the new
    :meth:`~scrapy.commands.ScrapyCommand.complete_argument` and
    :meth:`~scrapy.commands.ScrapyCommand.complete_option` methods. It replaces
    the completion scripts that were in the :file:`extras` directory of the
    source code repository.
    (:gh:`6453`, :gh:`7899`)

-   Added a :setting:`STICKY_META_KEYS` setting to copy request metadata keys
    from a response into the follow-up requests yielded by its callback,
    handled by the new
    :class:`~scrapy.spidermiddlewares.stickymeta.StickyMetaParamsMiddleware`.

    The new middleware has priority 1000 in
    :setting:`SPIDER_MIDDLEWARES_BASE`, and
    :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`
    moved from 1000 to 999.

    (:gh:`3645`, :gh:`3770`, :gh:`4141`, :gh:`7711`, :gh:`8166`)

-   :class:`~scrapy.spiders.Rule` has new ``name`` and ``from_rules``
    parameters, to apply a rule only to responses reached through specific
    other rules.
    (:gh:`982`, :gh:`8066`)

-   A callback docstring can now define several batches of :ref:`contracts
    <topics-contracts>`, one per paragraph, and a batch can repeat ``@url`` to
    check the callback against more than one sample request. Custom contracts
    can set the new ``generates_request`` attribute of
    :class:`~scrapy.contracts.Contract` to ``True`` to behave like ``@url``.
    (:gh:`768`, :gh:`8043`)

-   The :ref:`Google Cloud Storage feed storage <topics-feed-storage-gcs>` now
    supports the ``"append"`` :ref:`feed mode <feed-mode>`, instead of logging
    a warning and overwriting the feed.
    (:gh:`7823`, :gh:`8173`)

-   :setting:`FILES_EXPIRES` and :setting:`IMAGES_EXPIRES` now support
    fractional numbers of days, e.g. ``0.5`` for 12 hours, and negative values,
    which disable expiration, so that files are never downloaded again once
    fetched.
    (:gh:`2309`, :gh:`4731`, :gh:`8046`)

-   The *reactor_path* parameter of
    :func:`~scrapy.utils.reactor.install_reactor` is now optional, and
    defaults to the asyncio reactor.
    (:gh:`8125`)

-   Added a :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` setting which,
    when ``True``, passes exceptions raised by the ``process_response()``
    method of a downloader middleware to the ``process_exception()`` methods of
    the downloader middlewares that have not processed the response yet. This
    allows :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` to
    retry responses that other middlewares reject, e.g. responses that cannot
    be decompressed, for which :exc:`~scrapy.exceptions.DecompressionError` is
    now part of :setting:`RETRY_EXCEPTIONS`.
    (:gh:`1900`, :gh:`3506`, :gh:`7790`, :gh:`8177`)

-   :exc:`~scrapy.exceptions.CloseSpider` can now be raised from
    :ref:`downloader middlewares <topics-downloader-middleware>` to close the
    spider.
    (:gh:`2578`, :gh:`8177`)

-   Added :class:`JsonValidationMiddleware
    <scrapy.downloadermiddlewares.jsonvalidation.JsonValidationMiddleware>`,
    with priority 560 in :setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled
    with the :setting:`JSONVALIDATION_ENABLED` setting, which raises
    :exc:`json.JSONDecodeError` for successful
    :class:`~scrapy.http.JsonResponse` responses with an invalid JSON body.
    :exc:`json.JSONDecodeError` is now part of :setting:`RETRY_EXCEPTIONS`, so
    those responses are retried if
    :setting:`DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS` is ``True``.
    (:gh:`8215`)

-   Added an *error* keyword-only parameter to
    :exc:`~scrapy.exceptions.CloseSpider` and
    :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`, to mark a
    crawl as failed. Failed crawls set the new :stat:`finish_reason_error` stat
    to ``True``, get ``error=True`` in their :signal:`spider_closed` signal,
    and make the :command:`crawl` and :command:`runspider` commands exit with
    code 1. Crawls closed because :meth:`~scrapy.Spider.start` raised an
    exception (``start_error``) are now also marked as failed, and so are
    crawls closed by :setting:`CLOSESPIDER_ERRORCOUNT`.
    (:gh:`1241`, :gh:`2171`, :gh:`3653`, :gh:`3654`, :gh:`8082`, :gh:`8203`,
    :gh:`8204`, :gh:`8352`, :gh:`8357`)

-   Added an :setting:`IMAGES_PRESERVE_FORMAT` setting to :ref:`keep images in
    their original format <preserving-image-format>` instead of converting them
    to JPEG. Stored images then get no file extension.
    (:gh:`1705`, :gh:`2471`, :gh:`8198`)

-   Added an ``item_processor`` :ref:`feed option <feed-options>` to
    :ref:`replace each item with any number of items <item-processor>` in a
    given feed, and a ``feedexport/item_count/<storage class>`` stat with the
    number of exported entries per storage class.
    (:gh:`5905`, :gh:`7911`)

-   Added an :setting:`UPLOAD_TIMEOUT` setting to set how long uploads to
    remote :ref:`feed storages <topics-feed-storage-backends>` and :ref:`media
    pipeline storages <topics-media-pipeline>` (Amazon S3, Google Cloud Storage
    and FTP) wait for a response before giving up.
    (:gh:`5011`, :gh:`8162`)

-   Added a :setting:`MEDIA_CACHE_SIZE` setting to limit how many download
    outcomes :ref:`media pipelines <topics-media-pipeline>` keep in memory,
    which by default they keep for the whole crawl.
    (:gh:`939`, :gh:`8193`)

-   :setting:`LOG_FORMAT` now supports a ``%(spider)s`` placeholder with the
    name of the spider that triggered the log message, or ``-`` for messages
    not tied to a spider.
    (:gh:`1576`, :gh:`8086`)

-   :class:`~scrapy.statscollectors.StatsCollector` objects are now
    :class:`mutable mappings <collections.abc.MutableMapping>` of stats, e.g.
    ``stats["hostname"] = socket.gethostname()``.
    (:gh:`3230`, :gh:`8241`)

-   The :meth:`~scrapy.exporters.BaseItemExporter.start_exporting`,
    :meth:`~scrapy.exporters.BaseItemExporter.export_item` and
    :meth:`~scrapy.exporters.BaseItemExporter.finish_exporting` methods of
    :ref:`item exporters <topics-exporters>` used by :ref:`feed exports
    <topics-feed-exports>`, and the
    :meth:`~scrapy.extensions.feedexport.FeedStorageProtocol.open` and
    :meth:`~scrapy.extensions.feedexport.FeedStorageProtocol.store` methods of
    :ref:`feed storages <topics-feed-storage-custom>`, can now be coroutine
    functions.
    (:gh:`5622`, :gh:`6273`, :gh:`7800`)

-   Added a :setting:`LOG_LEVELS` setting, to set the minimum log level of
    specific loggers, overriding the levels that Scrapy sets by default.
    (:gh:`2231`, :gh:`7921`)

-   The :command:`settings` command now takes the settings of enabled
    :ref:`add-ons <topics-addons>` into account, and has a new ``--spider``
    option to take the settings of a spider into account as well.
    (:gh:`7010`, :gh:`7828`)

-   Added a :reqmeta:`skip_dupefilter_once` request meta key, which skips
    duplicate filtering for a request without affecting the requests derived
    from it, and a
    :attr:`~scrapy.core.scheduler.BaseScheduler.supports_skip_dupefilter_once`
    attribute for schedulers to declare that they support it.
    (:gh:`6895`, :gh:`8224`)

-   The FTP download handlers now list the contents of a directory when the
    URL path is empty or ends in ``/``, returning the output of the FTP
    ``NLST`` command, one name per line.
    (:gh:`1478`, :gh:`2505`, :gh:`8197`)

-   The :ref:`memory usage extension <topics-extensions-ref-memusage>` now
    works on Windows, with the new ``memusage`` :ref:`extra <extras>`.
    (:gh:`4070`, :gh:`8062`)

-   Added :class:`~scrapy.core.engine.EngineState` and
    :attr:`ExecutionEngine.state <scrapy.core.engine.ExecutionEngine.state>`,
    which tell the lifecycle state of the engine.
    (:gh:`6916`, :gh:`8217`)

-   The :command:`fetch`, :command:`view`, :command:`shell` and
    :command:`parse` commands now support a ``--curl`` option, to build the
    request from a cURL command instead of from a URL argument.
    (:gh:`36`, :gh:`447`, :gh:`7703`)

-   Added an experimental FTP and FTPS download handler,
    :class:`~scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler`,
    used by default when running :ref:`without a Twisted reactor
    <asyncio-without-reactor>`. The new ``socks`` :ref:`extra <extras>`
    enables its support for SOCKS proxies.
    (:gh:`7261`, :gh:`8213`, :gh:`8341`)

-   Added :setting:`MEDIA_STORAGES` and :setting:`MEDIA_STORAGES_BASE`
    settings, which map URL schemes to the storage classes of the :ref:`media
    pipelines <topics-media-pipeline>`, to support additional schemes or
    replace built-in storages. Storage classes can define a ``from_crawler()``
    class method to read settings, including settings specific to the files or
    images pipeline, or to a pipeline subclass.

    Pipelines no longer share the settings of built-in storages, so that e.g.
    the files and images pipelines of a crawl can use different values for
    :setting:`FILES_STORE_S3_ACL` and :setting:`IMAGES_STORE_S3_ACL`.
    (:gh:`5991`, :gh:`7925`)

-   Added :setting:`FILES_CHECKSUM_ALGORITHM` and
    :setting:`IMAGES_CHECKSUM_ALGORITHM` settings, to calculate the checksums
    of :ref:`media pipeline <topics-media-pipeline>` files with a
    :mod:`hashlib` algorithm other than MD5.
    (:gh:`3306`, :gh:`7951`)

-   :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor` has a new
    ``schemes`` parameter to choose the URL schemes of extracted links, and
    :data:`scrapy.linkextractors.SUPPORTED_SCHEMES` lists the URL schemes that
    Scrapy can download out of the box.
    (:gh:`4979`, :gh:`7994`)

-   Added :meth:`SitemapSpider.sitemap_request()
    <scrapy.spiders.SitemapSpider.sitemap_request>`, which you can override to
    customize the requests that :class:`~scrapy.spiders.SitemapSpider` builds
    from sitemap entries, e.g. to pass sitemap data to callbacks.
    (:gh:`2565`, :gh:`3543`, :gh:`8175`)

-   Added a :meth:`~scrapy.robotstxt.RobotParser.sitemaps` method to
    :class:`~scrapy.robotstxt.RobotParser`, implemented by the
    :ref:`Protego <protego-parser>` and :ref:`Robotexclusionrulesparser
    <rerp-parser>` parsers, which returns the sitemap URLs declared in the
    :file:`robots.txt` file, and a *response* parameter to the
    :signal:`robots_parsed` signal.
    (:gh:`3967`, :gh:`8069`, :gh:`8175`)

-   Added the :signal:`spider_state_loaded`, :signal:`spider_state_saving`
    and :signal:`feed_slots_initialized` signals.
    (:gh:`5153`, :gh:`7650`, :gh:`7656`)

-   Added an :ref:`event loop lag monitor <event-loop-lag>` extension, enabled
    by default, which logs a warning when code blocks the event loop for longer
    than the new :setting:`EVENT_LOOP_LAG_THRESHOLD` setting, 10 seconds by
    default.
    (:gh:`3197`, :gh:`8093`)

-   When :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
    denies at least one :ref:`start request <start-requests>` and no start
    request can be crawled, the crawl now finishes with the
    ``robotstxt_denied`` :stat:`finish_reason` instead of ``finished``, and an
    error is logged.
    (:gh:`6164`)

-   :func:`~scrapy.utils.httpobj.urlparse_cached` now also accepts a URL
    string, and ``scrapy.utils.url.url_is_from_any_domain()``,
    ``url_is_from_spider()`` and ``url_has_any_extension()`` now also accept
    :class:`~scrapy.Request` and :class:`~scrapy.http.Response` objects.
    Built-in components now reuse URL parsing results more, which makes
    crawls faster.
    (:gh:`8247`)

-   Added a :reqmeta:`proxy_headers` request meta key, to send headers to the
    proxy set in :reqmeta:`proxy`, e.g. in the ``CONNECT`` request of HTTPS
    URLs.
    (:gh:`3329`, :gh:`7888`)

-   Added
    :class:`~scrapy.downloadermiddlewares.uriuserinfo.UriUserInfoMiddleware`,
    enabled by default, which removes the credentials from request URLs like
    ``https://user:[email protected]`` and sets them as :reqmeta:`http_user`
    and :reqmeta:`http_pass`, or as :reqmeta:`ftp_user` and
    :reqmeta:`ftp_password` for FTP URLs. As a result, :attr:`Request.url
    <scrapy.Request.url>` and :attr:`Response.url <scrapy.http.Response.url>`
    no longer include those credentials after that middleware.
    (:gh:`669`, :gh:`851`, :gh:`1466`, :gh:`4474`)

-   Added an :setting:`HTTPCACHE_SCOPE` setting, which allows sharing the HTTP
    cache across spiders and, with
    :class:`~scrapy.extensions.httpcache.FilesystemCacheStorage`, splitting it
    by request host instead of by spider.
    (:gh:`5166`, :gh:`5175`, :gh:`7964`)

-   Added a :setting:`LOG_ROOT_FORMATTER` setting, which sets the formatter of
    the root log handler. Such formatters implement the new
    :class:`~scrapy.LogRootFormatterProtocol`, and the new
    :class:`~scrapy.LogRootFormatter` is the default one, which you can also
    use in your own log handlers.
    :setting:`LOG_COLOR` also works with custom formatters.
    (:gh:`8336`)

-   Added a :setting:`CONNECTION_KEEPALIVE_TIMEOUT` setting, which sets how
    long the built-in HTTP download handlers keep idle connections open for
    reuse. Its default value, 240 seconds, is the timeout that the
    Twisted-based handlers already used; the aiohttp- and httpx-based handlers
    used their library defaults, 15 and 5 seconds respectively.
    (:gh:`8342`)

-   Added :meth:`Response.follow_redirect()
    <scrapy.http.Response.follow_redirect>`, which returns a request that
    follows the ``Location`` header of the response.
    (:gh:`8331`)

-   When :setting:`DOWNLOAD_FAIL_ON_DATALOSS` or the
    :reqmeta:`download_fail_on_dataloss` request meta key is ``False``,
    hitting the download timeout while receiving the response body now
    returns the part of the response received so far, with the ``dataloss``
    flag, instead of raising :exc:`~scrapy.exceptions.DownloadTimeoutError`,
    with all built-in download handlers except
    :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`.
    (:gh:`3407`, :gh:`3492`, :gh:`8084`, :gh:`8464`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    now supports proxies with an IPv6 address, proxies whose host name only
    resolves to IPv6 addresses, given an IPv6-capable
    :setting:`TWISTED_DNS_RESOLVER`, and HTTPS requests to IPv6 addresses
    through a proxy.
    (:gh:`6674`, :gh:`7090`, :gh:`8116`, :gh:`8343`)

-   When asyncio support is available and ``aioboto3`` is installed,
    :ref:`S3 feed storage <topics-feed-storage-s3>` uploads files with it,
    instead of running ``boto3`` in a thread. Similarly, the :ref:`S3 storage
    of media pipelines <media-pipelines-s3>` uses ``aiobotocore``, when
    installed, instead of ``botocore``.
    (:gh:`3845`, :gh:`8330`, :gh:`8339`)

-   Added a :setting:`CONCURRENT_CONNECTIONS_PER_HANDLER` setting, which limits
    the number of connections, idle ones included, that each download handler
    keeps open. By default, the limit is derived from the maximum number of
    open files allowed for the process, or from the :func:`select.select`
    limit on Windows, to prevent ``Too many open files`` errors (see
    :ref:`faq-too-many-open-files`).
    (:gh:`2653`, :gh:`5024`, :gh:`5618`, :gh:`7932`)

-   Added a :setting:`REDIRECT_HTTP_CODES` setting and a
    :reqmeta:`redirect_http_codes` request meta key, which set the response
    status codes whose ``Location`` header
    :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` follows,
    e.g. to follow the ``Location`` header of 201 responses. Media pipelines
    with :setting:`MEDIA_ALLOW_REDIRECTS` enabled leave responses with those
    status codes to
    :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware`.
    (:gh:`1615`, :gh:`7801`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    supports ``https://`` proxy URLs for ``https://`` destination URLs.
    (:gh:`5828`, :gh:`7776`)

-   When several requests for the same resource are sent concurrently,
    :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now
    downloads it only once, and, if the response is cached, serves it from the
    cache to the other requests.
    The new :stat:`httpcache/wait` stat counts the requests that waited for
    another one.
    (:gh:`2744`, :gh:`7915`)

-   ``XMLFeedSpider.parse_node()``, ``CSVFeedSpider.parse_row()`` and the
    *errback* of :class:`~scrapy.spiders.Rule` can now be defined as coroutine
    functions (``async def``) or asynchronous generator functions.
    (:gh:`8392`)

-   ``item_completed()`` in subclasses of ``MediaPipeline`` can now be defined
    as a coroutine function (``async def``), and ``stat_file()`` in
    :ref:`custom media storages <media-pipeline-custom-storage>` can now await
    :mod:`asyncio` code.
    (:gh:`8413`)

-   :meth:`BaseSettings.getlist() <scrapy.settings.BaseSettings.getlist>` now
    parses a string that is a JSON list, e.g. ``-s 'MY_LIST=["a,b", "c"]'``, so
    that items that contain commas can be set from the command line.
    (:gh:`8416`)

-   Added a :setting:`RESPONSE_ROUGH_SIZE` setting, the
    :attr:`ExecutionEngine.response_active_size
    <scrapy.core.engine.ExecutionEngine.response_active_size>` attribute, and
    the :stat:`request_backout_seconds/concurrency`,
    :stat:`request_backout_seconds/response_max_active_size` and
    :stat:`request_backout_seconds/total` stats. See
    :ref:`crawl-bottlenecks`.
    (:gh:`5302`, :gh:`6206`, :gh:`6430`)

Improvements
~~~~~~~~~~~~

-   The encoding of a response that doesn't declare one is now detected the
    way web browsers do: UTF-8 if the body is valid UTF-8, otherwise a guess
    based on the body and on the top-level domain of the response URL, as
    Firefox does. Encoding declarations in ``<meta>`` tags are also found the
    way web browsers find them, and declared encoding names are mapped to
    encodings the way web browsers map them, e.g. ``gb2312`` to GBK instead of
    GB18030.
    (:gh:`8393`)

-   With parsel_ 1.12.0+, :class:`~scrapy.Selector` objects created from a
    UTF-8 response now parse the response body as is, instead of decoding it
    first, which saves time and memory.
    (:gh:`5906`, :gh:`8195`, :gh:`8222`, :gh:`8393`)

-   Responses are now freed as soon as nothing refers to them, also after
    using their selector or after their callback raises an exception, instead
    of when the garbage collector runs, which lowers peak memory usage.
    (:gh:`2977`, :gh:`8245`)

-   The reactor settings :setting:`DNSCACHE_ENABLED`, :setting:`DNSCACHE_SIZE`,
    :setting:`DNS_TIMEOUT`, :setting:`REACTOR_THREADPOOL_MAXSIZE` and
    :setting:`TWISTED_DNS_RESOLVER` can now be :ref:`set from a spider
    <spider-settings>` when using :class:`~scrapy.crawler.CrawlerProcess` or
    :class:`~scrapy.crawler.AsyncCrawlerProcess`. When :ref:`running multiple
    spiders in the same process <run-multiple-spiders>`, the values of the
    first spider are used, and a warning is issued for any later spider that
    defines a different value. The DNS resolver now gets the
    :class:`~scrapy.crawler.Crawler` of that first spider in its
    ``from_crawler()`` method.
    (:gh:`5988`, :gh:`7923`)

-   The :command:`runspider` command now accepts file paths without an
    extension, such as the ones of Bash process substitution
    (``scrapy runspider <(…)``).
    (:gh:`4827`, :gh:`8036`)

-   Items are now processed by tasks started on demand, instead of by a fixed
    number of tasks, one per :setting:`CONCURRENT_ITEMS`, started for every
    response, so a high :setting:`CONCURRENT_ITEMS` value no longer slows down
    crawls. :setting:`CONCURRENT_ITEMS` values lower than ``1`` now work as
    ``1``.
    (:gh:`5182`, :gh:`7959`)

-   Exceptions raised while exporting an item to a :ref:`feed
    <topics-feed-exports>` now include the item in a note.
    (:gh:`2775`, :gh:`8191`)

-   When a :class:`~scrapy.http.TextResponse` is created with a :class:`str`
    body and a UTF-8 encoding, that body is now reused as
    :attr:`~scrapy.http.TextResponse.text` instead of being decoded again.
    (:gh:`8220`)

-   Improved the performance of link extraction with
    :class:`~scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor`.
    (:gh:`8126`)

-   Compressed response bodies of 8 KiB or more are now decompressed in a
    thread, so that other responses can be processed meanwhile.
    (:gh:`3751`, :gh:`8153`)

-   The Scrapy version and the versions of :setting:`LOG_VERSIONS` are now
    logged when each crawler is set up, after :ref:`add-ons <topics-addons>`
    and spider settings are applied, so that they can change
    :setting:`BOT_NAME` and :setting:`LOG_VERSIONS`. They are now also logged
    when using :class:`~scrapy.crawler.CrawlerRunner` or
    :class:`~scrapy.crawler.AsyncCrawlerRunner`. Add-ons can now :ref:`log
    the versions of their dependencies <topics-addons>`.
    (:gh:`8246`)

-   ``scrapy.pipelines.files.S3FilesStore`` now supports any header that
    botocore_ supports for uploads, and :ref:`media pipelines
    <topics-media-pipeline>` now document how to set custom headers for files
    uploaded to Amazon S3.
    (:gh:`4788`, :gh:`4813`, :gh:`8171`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    sends the ``Content-Length`` request header that you set, instead of
    replacing it, with a warning, with one built from the request body.
    (:gh:`4919`, :gh:`7944`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    supports :setting:`DOWNLOAD_FAIL_ON_DATALOSS` and the
    :reqmeta:`download_fail_on_dataloss` request meta key: when the setting or
    meta key is ``False``, responses cut short by a lost connection are now
    returned with the ``dataloss`` flag; otherwise, they now fail with
    :exc:`~scrapy.exceptions.ResponseDataLossError` instead of
    :exc:`~scrapy.exceptions.DownloadFailedError`. HTTP/2 data loss is now also
    detected by
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
    (:gh:`5048`, :gh:`7989`)

-   Spiders that request the :file:`robots.txt` file of a host, like
    :class:`~scrapy.spiders.SitemapSpider`, no longer download it twice when
    :setting:`ROBOTSTXT_OBEY` is ``True``:
    :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
    parses the response of a request for the :file:`robots.txt` file of a
    host, if it has not downloaded it yet, instead of downloading it again.
    (:gh:`3967`, :gh:`8069`)

-   Requests that set the :reqmeta:`download_maxsize` request meta key but not
    :reqmeta:`download_warnsize` no longer log a warning based on
    :setting:`DOWNLOAD_WARNSIZE`.
    (:gh:`8213`)

-   :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    is now :ref:`lazy <lazy-download-handlers>`.
    (:gh:`8346`)

-   Connecting a handler to a built-in signal, where the handler declares an
    argument which the signal never sends, e.g. because of a typo, now issues a
    warning.
    (:gh:`8345`)

-   :func:`~scrapy.utils.asyncio.call_later` and the processing of the output
    of spider callbacks now run in the :mod:`contextvars` context that was
    current when they were scheduled also when not using the asyncio reactor,
    so that context variables set by your code, e.g. for tracing or logging,
    are kept.
    (:gh:`8347`)

-   Improved the performance of ``scrapy.utils.python.binary_is_text()``,
    which is used to pick the class of responses.
    (:gh:`8362`)

-   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now
    ignores received cookies with a ``__Secure-`` or ``__Host-`` name prefix
    that do not meet the requirements of that prefix, as web browsers do, e.g.
    such cookies received over plain HTTP.
    (:gh:`5431`, :gh:`8431`)

-   Improved the performance of :ref:`feed exports <topics-feed-exports>`,
    especially for items with scalar field values.
    (:gh:`8461`)

-   Improved the performance of :class:`~scrapy.statscollectors.StatsCollector`
    method calls.
    (:gh:`8243`)

-   Other code refactoring and improvements.
    (:gh:`8262`, :gh:`8263`, :gh:`8295`, :gh:`8313`, :gh:`8323`, :gh:`8335`,
    :gh:`8353`, :gh:`8365`, :gh:`8385`, :gh:`8391`, :gh:`8415`)

Bug fixes
~~~~~~~~~

-   :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now raises
    :exc:`~scrapy.exceptions.DownloadFailedError` when the server drops the
    connection, so that those downloads are retried.
    (:gh:`1511`, :gh:`8206`)

-   :class:`~scrapy.pipelines.files.FilesPipeline` now treats responses with
    the 201 status code as successful downloads.
    (:gh:`8349`)

-   The ``Cookie`` header now lists cookies with the same path length from
    oldest to newest, as :rfc:`6265#section-5.4` requires.
    (:gh:`5431`, :gh:`8321`)

-   The :command:`runspider` command now runs the given file even if its name
    matches that of an already imported module, e.g. :file:`json.py`.
    (:gh:`8036`, :gh:`8463`)

-   The :command:`check` command now reports errors that prevent a spider
    from crawling, e.g. errors raised while creating the spider, instead of
    silently ignoring them.
    (:gh:`3716`, :gh:`7991`)

-   :meth:`Request.from_curl() <scrapy.Request.from_curl>` now handles
    headers without a value (``-H 'Name;'``) as curl does, ignores headers
    that curl would not send, supports ``-u`` with a user name and no
    password, and raises :exc:`ValueError` for an empty command, instead of
    failing with unexpected exceptions. It also turns ``-H 'Name:'`` into a
    header set to ``None``, i.e. not sent, instead of a header with an empty
    value, accepts Bash ``$'…'`` quoting in any argument, e.g. headers,
    although it does not decode backslash escapes, and keeps a leading ``$`` in
    request data instead of removing it.

    :meth:`Request.to_curl() <scrapy.Request.to_curl>` now includes every
    value of repeated headers, handles headers set to ``None`` and no longer
    fails on headers that are not valid UTF-8.

    (:gh:`8131`, :gh:`8138`, :gh:`8148`, :gh:`8267`, :gh:`8291`, :gh:`8292`)

-   ``Headers.to_tuple_list()`` no longer raises :exc:`UnicodeDecodeError`;
    bytes that cannot be decoded are replaced with U+FFFD.
    (:gh:`2592`, :gh:`2637`, :gh:`8192`)

-   :func:`copy.copy` on an :class:`~scrapy.Item` now returns a shallow copy,
    like :meth:`Item.copy() <scrapy.Item.copy>`. It used to return an item
    that shared its fields with the original, so that setting or deleting a
    field on one also did it on the other.
    (:gh:`8172`)

-   Calling ``exit()`` or ``quit()`` in the :ref:`telnet console
    <topics-telnetconsole>` now closes the connection, instead of logging an
    unhandled error.
    (:gh:`3853`, :gh:`8065`)

-   An exception raised while building the :setting:`SCHEDULER` is now
    reported as is, instead of being followed by a
    :exc:`RuntimeError` about an unassigned engine slot.
    (:gh:`8169`)

-   The ``lastseen`` field of the representation of a downloader slot, e.g.
    in the :ref:`telnet console <topics-telnetconsole>`, now shows the
    monotonic timestamp of the last activity instead of a meaningless date.
    (:gh:`8182`)

-   :class:`~scrapy.spiders.SitemapSpider` now resolves relative and
    protocol-relative URLs in sitemaps against the sitemap URL.
    (:gh:`965`, :gh:`8210`)

-   :class:`~scrapy.spiders.CrawlSpider` now follows the links of a response
    also when the callback of the matching rule raises an exception.
    (:gh:`4108`, :gh:`8200`)

-   A :ref:`feed <topics-feed-exports>` URI with a placeholder that is neither
    a spider attribute nor returned by the ``uri_params`` function now makes
    only that feed be skipped, with an error logged, instead of raising
    :exc:`KeyError` and leaving the remaining feeds unopened.
    (:gh:`2089`, :gh:`8205`)

-   Fixed the file modification times that
    :class:`~scrapy.pipelines.files.FilesPipeline` reads from Amazon S3, Google
    Cloud Storage and FTP storages to apply :setting:`FILES_EXPIRES`. S3 and
    Google Cloud Storage times were off by the UTC offset of the local time
    zone, and FTP times were misread.
    (:gh:`8186`, :gh:`8196`)

-   Lines of a callback docstring that start with ``@`` but do not name a
    registered :ref:`contract <topics-contracts>`, e.g. a decorator in a code
    example, are now ignored, instead of raising :exc:`KeyError` or making the
    :command:`check` command treat the callback as having contracts. As a
    result, a misspelled contract name is also ignored.
    (:gh:`8226`)

-   The :command:`bench` command now listens on a random free port, instead of
    failing when port 8998 is in use.
    (:gh:`8230`)

-   :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and the ``concurrency`` key of
    :setting:`DOWNLOAD_SLOTS` are now enforced when using asyncio with no
    download delay.
    (:gh:`8256`, :gh:`8257`)

-   :class:`~scrapy.spiders.SitemapSpider` now detects XML sitemaps also in
    responses that are not :class:`~scrapy.http.XmlResponse` objects, e.g.
    from third-party download handlers, based on their headers, URL and body.
    (:gh:`6362`, :gh:`7904`)

-   Fixed the handling of headers that are not valid UTF-8 or ASCII in some
    download handlers:

    - :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`
      no longer fails on non-ASCII request headers.

    - :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
      no longer fails on request or response headers that are not valid
      UTF-8. Invalid UTF-8 in request headers is sent as U+FFFD, with a
      warning.

    - :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no
      longer fails, or hangs, on request or response headers that are not
      valid UTF-8.

    (:gh:`8265`)

-   ``Headers.get()`` now returns the given default value for a header set to
    ``None``.
    (:gh:`8268`)

-   Fixed the ``downloader/request_bytes`` and ``downloader/response_bytes``
    stats, which miscounted line breaks for requests with all headers set to
    ``None`` and for responses with multi-value headers, and
    ``scrapy.utils.request.request_httprepr()``, which added an extra line
    break for requests with all headers set to ``None``.
    (:gh:`8266`, :gh:`8271`)

-   Changing the header values in the output of :meth:`Request.to_dict()
    <scrapy.Request.to_dict>` or :meth:`Response.to_dict()
    <scrapy.http.Response.to_dict>` no longer changes the original request or
    response.
    (:gh:`8269`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    closes the proxy connection of an HTTPS request that times out, or is
    otherwise cancelled, before the proxy has established the tunnel.
    (:gh:`3614`, :gh:`7810`)

-   :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` no
    longer copies the fragment of the original URL into the redirect URL when
    the ``Location`` header ends with an empty fragment (``#``).
    (:gh:`8100`)

-   When using a :ref:`job directory <job-dir>`, the default scheduler now pops
    requests by :attr:`~scrapy.Request.priority` across its memory and disk
    queues, instead of popping every request from the memory queue before any
    from the disk queue.

    Custom :setting:`SCHEDULER_PRIORITY_QUEUE` classes need to implement the
    new :meth:`~scrapy.pqueues.PriorityQueueProtocol.next_priority` method to
    get this behavior. :class:`~scrapy.pqueues.PriorityQueueProtocol`
    documents the interface of priority queue classes.
    (:gh:`3666`, :gh:`8237`)

-   When resuming a crawl from a :ref:`job directory <job-dir>`, feeds whose
    URI uses ``%(batch_id)d`` now continue the batch numbering of the previous
    run, instead of starting from 1 again and overwriting the existing files.
    (:gh:`5153`, :gh:`7650`, :gh:`7656`)

-   :ref:`Item exporters <topics-exporters>` now serialize nested items, and
    lists of them, using the serializers declared in their own fields.
    (:gh:`3888`, :gh:`3890`, :gh:`8057`)

-   Fixed :class:`~scrapy.utils.asyncio.AsyncioLoopingCall` sometimes calling
    its function twice in a row when the event loop woke it up slightly early,
    which could, for example, close a spider because of
    :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM` while it was still producing items.
    (:gh:`8294`)

-   The :ref:`RobotFileParser <python-robotfileparser>` parser now ignores
    the invalid bytes of a :file:`robots.txt` file that is not valid UTF-8,
    like the other built-in parsers, instead of ignoring the whole file.
    (:gh:`8316`)

-   Exceptions raised while handling the output of a spider callback, e.g.
    by a custom scheduler or log formatter, are now logged, and no longer
    interfere with the handling of the rest of that output.
    (:gh:`8318`)

-   Scrapy now uses the event loop of the installed asyncio reactor when no
    event loop is running, instead of the current event loop of the thread,
    which can be a different one or none at all, e.g. when the reactor was
    installed with a loop that is not set as the current one.
    (:gh:`7201`, :gh:`8324`, :gh:`8325`)

-   Stopping a crawl while an item yielded from :meth:`~scrapy.Spider.start` is
    going through the :ref:`item pipelines <topics-item-pipeline>` no longer
    hangs.
    (:gh:`8348`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    fails requests with the connection error when it cannot connect to the
    server, instead of letting them hit the download timeout, and later
    requests to the same server try to connect again.
    (:gh:`8332`)

-   Fixed :class:`~scrapy.pqueues.ScrapyPriorityQueue`, which the default
    :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` uses for each
    download slot, returning no request or a request other than the one that
    its ``peek()`` method returned, which could happen with requests of the
    same priority in its start request and regular request queues, or after a
    request failed to be serialized into a disk queue.
    (:gh:`7837`)

-   The :ref:`telnet console <topics-telnetconsole>`, and any other caller of
    ``scrapy.utils.reactor.listen_tcp()``, now tries the next port when another
    process starts listening on the same port at the same time, instead of
    failing to start.
    (:gh:`8356`)

-   Fixed :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`
    letting through requests disallowed by :file:`robots.txt` when they were
    sent while the :file:`robots.txt` file of their website was being
    downloaded for another request.
    (:gh:`8297`, :gh:`8386`)

-   :class:`~scrapy.spiders.SitemapSpider` now only treats links with
    ``rel="alternate"`` as alternate links, instead of any link, e.g.
    canonical ones, both when
    :attr:`~scrapy.spiders.SitemapSpider.sitemap_alternate_links` is enabled
    and in the entries passed to
    :meth:`~scrapy.spiders.SitemapSpider.sitemap_filter`.
    (:gh:`8390`)

-   Download handlers are now closed when the engine is closed without having
    been started, e.g. when opening the spider fails, and the downloader is now
    closed when the engine is stopped without a spider.
    (:gh:`8167`, :gh:`8377`)

-   The spider is no longer considered idle, and so no longer closed, while a
    request taken from the scheduler has not reached the downloader yet, which
    could happen with a custom :setting:`DOWNLOADER`.
    (:gh:`8315`, :gh:`8355`)

-   The reactorless :command:`shell` now waits for the engine to start before
    letting you use it.
    (:gh:`8168`)

-   On Windows,
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    no longer connects to the wrong port in rare cases.
    (:gh:`8382`)

-   The Scrapy :command:`shell` no longer uses ptpython_ on Windows when the
    standard output is not a console, which ptpython_ does not support.
    (:gh:`8367`)

Documentation
~~~~~~~~~~~~~

-   Documented :ref:`how blocking code in callbacks and other components
    slows down downloads <optimize-blocking>` and inflates their
    :reqmeta:`download_latency`.
    (:gh:`3197`, :gh:`8160`)

-   Documented :ref:`how to change concurrency and delays from the telnet
    console <telnet-concurrency>` on a running crawl.
    (:gh:`4196`, :gh:`8164`)

-   Documented that the ``open_spider()`` and ``close_spider()`` methods of
    :ref:`item pipelines <topics-item-pipeline>` run concurrently, and only
    ``process_item()`` follows the :setting:`ITEM_PIPELINES` order.
    (:gh:`2645`, :gh:`8158`)

-   Documented :ref:`how to set up a development environment <dev-setup>` to
    contribute to Scrapy.
    (:gh:`4451`, :gh:`8165`)

-   Moved the release notes of Scrapy 2.13 and older into separate pages per
    major version.
    (:gh:`8150`)

-   Documented :ref:`how to extract clean content and parse values
    <extraction>`, :ref:`how to sign requests with OAuth <mw-oauth>`, how to
    deduplicate repeated query string parameters in :ref:`request fingerprints
    <custom-request-fingerprinter>`, and that retried status codes should also
    be in :setting:`HTTPCACHE_IGNORE_HTTP_CODES`.
    (:gh:`40`, :gh:`1113`, :gh:`2297`, :gh:`8190`, :gh:`8202`, :gh:`8208`,
    :gh:`8209`)

-   Improved the API reference of :class:`~scrapy.spiders.SitemapSpider`,
    :class:`~scrapy.spiders.XMLFeedSpider` and
    :class:`~scrapy.spiders.CSVFeedSpider`.
    (:gh:`8229`, :gh:`8239`)

-   Documented the order of the :signal:`spider_opened`,
    :signal:`engine_started`, :signal:`spider_closed` and
    :signal:`engine_stopped` signals.
    (:gh:`1523`, :gh:`8101`)

-   Documented that the :setting:`JOBDIR` and :setting:`HTTPCACHE_DIR`
    directories must be protected from :ref:`write access by untrusted
    parties <security-local-data>`.
    (:gh:`8252`)

-   Documented :ref:`how to write custom feed storage backends
    <topics-feed-storage-custom>` and the :ref:`built-in feed storage backend
    classes <feed-storage-classes>`.
    (:gh:`7800`)

-   Added an introduction video to the :ref:`page about coding agents
    <agents>`.
    (:gh:`8253`)

-   :class:`~scrapy.spiders.CrawlSpider` and :class:`~scrapy.spiders.Rule` are
    now documented through their docstrings, and the documentation now
    recommends :class:`~scrapy.Spider` over
    :class:`~scrapy.spiders.CrawlSpider` for crawls that do not fit its
    declarative rules.
    (:gh:`7025`, :gh:`8240`)

-   Documented the
    :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`,
    :meth:`~scrapy.core.engine.ExecutionEngine.start_async`,
    :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` and
    :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` methods of the
    execution engine.
    (:gh:`6164`, :gh:`8462`)

-   Documented :ref:`how to write asyncio code that also supports older Scrapy
    versions <asyncio-old-versions>`, and the documentation now recommends
    :func:`asyncio.to_thread` to call blocking code from :ref:`coroutines
    <topics-coroutines>`.
    (:gh:`7956`, :gh:`8454`)

-   Reviewed the documentation of :class:`~scrapy.Spider`, which now covers
    all its attributes, including :attr:`~scrapy.Spider.state`.
    (:gh:`7025`, :gh:`7811`)

-   Updated the :ref:`installation guide <intro-install>`, and documented
    :ref:`what to do when installing a dependency fails <install-no-wheel>`.
    (:gh:`5308`, :gh:`7878`)

-   The :ref:`tutorial <intro-tutorial>` now shows how to crawl
    ``quotes.toscrape.com`` faster with :setting:`DOWNLOAD_SLOTS`.
    (:gh:`8334`)

-   Documented :exc:`~scrapy.spidermiddlewares.httperror.HttpError`, the
    IPv6 support of the built-in :ref:`download handlers
    <topics-download-handlers>`, the ``bootstrap_failed`` attribute of crawler
    processes, and where exceptions that interrupt a crawl started from a
    :ref:`script <run-from-script>` reach your code.
    (:gh:`8338`, :gh:`8340`, :gh:`8344`)

-   Added a :ref:`page about getting help <getting-help>`.
    (:gh:`3688`)

-   Documented that high values of :setting:`CONCURRENT_REQUESTS` may exceed
    the limit of open file descriptors that the operating system sets, and how
    to raise that limit.
    (:gh:`5259`, :gh:`5272`)

-   Other documentation improvements and fixes.
    (:gh:`5859`, :gh:`7114`, :gh:`8151`, :gh:`8161`, :gh:`8314`, :gh:`8403`,
    :gh:`8404`, :gh:`8436`, :gh:`8453`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Improved and fixed type hints.
    (:gh:`7811`, :gh:`8078`, :gh:`8183`, :gh:`8311`, :gh:`8333`, :gh:`8440`)

-   Improved CodSpeed benchmarks.
    (:gh:`8227`, :gh:`8234`, :gh:`8456`)

-   Improved the documentation build and the contributor tooling, including
    an ``AGENTS.md`` file for coding agents.
    (:gh:`8011`, :gh:`8154`, :gh:`8157`, :gh:`8188`, :gh:`8212`, :gh:`8214`,
    :gh:`8216`, :gh:`8219`, :gh:`8238`, :gh:`8251`, :gh:`8364`, :gh:`8384`,
    :gh:`8409`, :gh:`8411`)

-   CI and test improvements and fixes.
    (:gh:`6141`, :gh:`7949`, :gh:`8127`, :gh:`8155`, :gh:`8156`, :gh:`8194`,
    :gh:`8223`, :gh:`8228`, :gh:`8231`, :gh:`8232`, :gh:`8233`, :gh:`8235`,
    :gh:`8236`, :gh:`8258`, :gh:`8273`, :gh:`8274`, :gh:`8275`, :gh:`8276`,
    :gh:`8277`, :gh:`8278`, :gh:`8279`, :gh:`8280`, :gh:`8281`, :gh:`8286`,
    :gh:`8300`, :gh:`8301`, :gh:`8311`, :gh:`8314`, :gh:`8317`, :gh:`8320`,
    :gh:`8351`, :gh:`8359`, :gh:`8360`, :gh:`8361`, :gh:`8372`, :gh:`8374`,
    :gh:`8375`, :gh:`8381`, :gh:`8387`, :gh:`8388`, :gh:`8389`, :gh:`8406`,
    :gh:`8412`, :gh:`8423`, :gh:`8444`)

.. _release-2.19.0:

Scrapy 2.19.0 (2026-09-10)
--------------------------

Highlights:

-   New ``RemoteControl`` extension which allows inspecting and controlling a
    running crawl over HTTP, used by the :ref:`Scrapy MCP server
    <using-mcp-server>`

-   Experimental ``aiohttp``-based download handler (now the default when
    running without a reactor)

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   Added support for Python 3.15.
    (:gh:`7511`)

-   New dependencies:

    - aiohttp_ >= 3.13.3

    - charset-normalizer_ >= 3.4.0

    - platformdirs_ >= 2.0.0

    (:gh:`7866`, :gh:`8054`)

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   When running :ref:`without a Twisted reactor <asyncio-without-reactor>`,
    i.e. with :setting:`TWISTED_REACTOR_ENABLED` set to ``False``, the default
    download handler for ``http`` and ``https`` is now
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    instead of
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. You
    can configure
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` in
    the :setting:`DOWNLOAD_HANDLERS` setting if you want.
    (:gh:`8118`)

-   ``scrapy.http.cookies.WrappedRequest.is_unverifiable()`` now always returns
    ``False``, and the undocumented ``is_unverifiable`` request meta key that
    it used to read is ignored.
    (:gh:`7933`)

Deprecations
~~~~~~~~~~~~

-   The ``RANDOMIZE_DOWNLOAD_DELAY`` setting is deprecated. Use the new
    :setting:`DOWNLOAD_DELAY_JITTER` setting instead. Similarly, the
    ``randomize_delay`` key of the :setting:`DOWNLOAD_SLOTS` setting is
    deprecated in favor of a new ``jitter`` key, the ``randomize_delay``
    attribute of :class:`scrapy.core.downloader.Slot` is deprecated in favor of
    its new ``jitter`` attribute, and the ``randomize_delay`` attribute
    of :class:`scrapy.core.downloader.Downloader` is deprecated in favor of the
    :setting:`DOWNLOAD_DELAY_JITTER` setting.
    (:gh:`7881`)

-   The ``install_root_handler`` parameter of
    :func:`~scrapy.utils.log.configure_logging`,
    :class:`~scrapy.crawler.CrawlerProcess` and
    :class:`~scrapy.crawler.AsyncCrawlerProcess` is deprecated. Use the new
    :setting:`LOG_INSTALL_ROOT_HANDLER` setting instead.
    (:gh:`4793`, :gh:`8041`)

-   The ``scrapy.dupefilters.RFPDupeFilter.fingerprints`` attribute is
    deprecated. Overriding
    :meth:`scrapy.dupefilters.RFPDupeFilter.request_fingerprint` is deprecated
    as well; set the :setting:`REQUEST_FINGERPRINTER_CLASS` setting instead.
    (:gh:`5517`, :gh:`7943`)

-   Overriding the ``add_pre_hook()`` or ``add_post_hook()`` methods of
    :class:`~scrapy.contracts.Contract` is deprecated. Define
    ``pre_process()`` or ``post_process()`` instead.
    (:gh:`6681`, :gh:`7886`)

New features
~~~~~~~~~~~~

-   Added a :class:`~scrapy.extensions.remote_control.RemoteControl`
    extension, enabled by default, which allows connecting to crawl processes
    via HTTP and running code inside them.
    (:gh:`7866`)

-   Added an experimental HTTP download handler based on aiohttp_,
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`.
    (:gh:`8118`)

-   Added SQLite-backed scheduler queues:
    :class:`~scrapy.squeues.PickleFifoSQLiteQueue`,
    :class:`~scrapy.squeues.PickleLifoSQLiteQueue`,
    :class:`~scrapy.squeues.MarshalFifoSQLiteQueue` and
    :class:`~scrapy.squeues.MarshalLifoSQLiteQueue`. They write each request
    within its own transaction, so that an unclean shutdown cannot corrupt the
    on-disk queue, at the cost of slower scheduling.
    (:gh:`845`, :gh:`7877`)

-   Added a :setting:`DOWNLOAD_DELAY_JITTER` setting, and a ``jitter`` key for
    :setting:`DOWNLOAD_SLOTS`, which set the magnitude of the random variation
    applied to :setting:`DOWNLOAD_DELAY`. They replace the
    ``RANDOMIZE_DOWNLOAD_DELAY`` setting and the ``randomize_delay`` key, which
    could only toggle a fixed ±50%.
    (:gh:`7881`)

-   Added a :setting:`LOG_COLOR` setting, which colorizes log output by log
    level when logging to a terminal. It needs the new :ref:`color <extras>`
    extra.
    (:gh:`8091`)

-   The encoding of a response that doesn't declare one is now detected with
    charset-normalizer_ when the response body is neither ASCII nor UTF-8,
    instead of always falling back to ``cp1252``.
    (:gh:`3135`, :gh:`8054`)

-   Added a :setting:`LOG_INSTALL_ROOT_HANDLER` setting, which replaces the
    deprecated ``install_root_handler`` parameter and, unlike it, can also be
    set from a :file:`settings.py` module or from the command line.
    (:gh:`4793`, :gh:`8041`)

-   :ref:`Contracts <topics-contracts>` now support callbacks defined with
    ``async def``, including asynchronous generators.
    (:gh:`6681`, :gh:`7886`)

-   Added the :class:`~scrapy.contracts.default.MethodContract` (``@method``),
    :class:`~scrapy.contracts.default.BodyContract` (``@body``),
    :class:`~scrapy.contracts.default.HeaderContract` (``@header``) and
    :class:`~scrapy.contracts.default.CookieContract` (``@cookie``)
    contracts, which set the corresponding attributes of the sample request,
    and an ``-a`` option for the :command:`check` command, to set spider
    arguments as in the :command:`crawl` command.
    (:gh:`1918`, :gh:`8053`)

-   Added :meth:`Response.to_dict() <scrapy.http.Response.to_dict>`,
    :meth:`Response.from_dict() <scrapy.http.Response.from_dict>` and
    :func:`~scrapy.utils.response.response_from_dict`, and used them in the
    built-in :ref:`HTTP cache storages <httpcache-storage-fs>`, which now
    restore cached responses of any response class, including those of
    third-party plugins, with all their attributes.
    (:gh:`1450`, :gh:`7908`)

-   Cached responses now indicate when they were stored, through the new
    :reqmeta:`cache_timestamp` request meta key.
    (:gh:`2221`, :gh:`8034`)

-   The ``format`` key of :setting:`FEEDS` is now inferred from the file
    extension of the feed URI when not set, e.g. ``json`` for a URI ending in
    :file:`.json`.
    (:gh:`1158`, :gh:`8031`)

-   Added :func:`~scrapy.utils.project.find_projects`, which yields the root
    directory of every Scrapy project in a directory tree.
    (:gh:`8024`)

-   Added a public :attr:`~scrapy.core.engine.ExecutionEngine.scheduler`
    attribute to :class:`~scrapy.core.engine.ExecutionEngine`.
    (:gh:`8099`)

-   :class:`~scrapy.statscollectors.StatsCollector` objects now implement
    ``__str__()``, which returns the pretty-printed stats.
    (:gh:`2746`)

-   :func:`~scrapy.utils.response.open_in_browser` now also supports
    responses that are neither HTML nor plain text, picking a file extension
    based on the ``Content-Type`` header.
    (:gh:`3902`, :gh:`8067`)

-   The ``-t``/``--template`` option of the :command:`genspider` command now
    also takes a path to a :file:`.tmpl` file, so that a custom template can
    be used without setting :setting:`TEMPLATES_DIR`.
    (:gh:`8071`)

-   The ``--pdb`` command-line option now uses ipdb_ instead of :mod:`pdb`
    when ipdb_ is installed.
    (:gh:`4284`, :gh:`8052`)

-   Log records about item processing, i.e. about items scraped, dropped or
    raising an exception, now carry the item in their ``extra`` dict, under
    the ``item`` key, so that custom logging handlers can read it.
    (:gh:`8085`)

Improvements
~~~~~~~~~~~~

-   :class:`~scrapy.dupefilters.RFPDupeFilter` now keeps request fingerprints
    as :class:`bytes`, to reduce memory usage. The :file:`requests.seen` file
    that it writes in the :ref:`job directory <job-dir>` is now a binary file.
    (:gh:`5517`, :gh:`7943`)

-   The ``--pdb`` command-line option now starts a post-mortem debugging
    session on every logged error that comes with a traceback, instead of on
    every :class:`~twisted.python.failure.Failure` object built anywhere in
    the process.
    (:gh:`3552`, :gh:`8037`)

-   The Scrapy :command:`shell` no longer prints the ``DEBUG`` messages of
    parso_, an indirect dependency of IPython, when :setting:`LOG_LEVEL` is
    ``DEBUG``.
    (:gh:`8060`)

-   Other code refactoring and improvements.
    (:gh:`8123`)

Bug fixes
~~~~~~~~~

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    now sets :attr:`~scrapy.http.Response.certificate` and
    :attr:`~scrapy.http.Response.ip_address` attributes even for responses
    without a body.
    (:gh:`4466`, :gh:`8048`)

-   HTTP connection pool keys in
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    include the request bind address (see :setting:`DOWNLOAD_BIND_ADDRESS`), so
    that requests bound to different addresses no longer reuse each other's
    pooled connections.
    (:gh:`3565`, :gh:`8081`)

-   :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now
    updates the headers of a cached response, and stores it again, when a
    revalidation request gets a 304 response.
    (:gh:`3778`, :gh:`8068`)

-   An exception raised in a spider callback that no
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
    method handles is now offered to each of those methods only once.
    (:gh:`4729`, :gh:`7996`)

-   Defining :attr:`~scrapy.Spider.allowed_domains` as a property no longer
    breaks the commands that match a URL to a spider, i.e. :command:`shell`,
    :command:`fetch` and :command:`parse`. A property cannot be evaluated on
    a spider class, so it is now ignored, with a warning, instead of raising
    :exc:`TypeError`.
    (:gh:`3119`, :gh:`8094`)

-   :func:`~scrapy.utils.project.get_project_settings` now adds the project
    directory to :data:`sys.path` also when the ``SCRAPY_SETTINGS_MODULE``
    environment variable is set, so that the settings module that the variable
    points to can be imported.
    (:gh:`4780`, :gh:`8042`)

-   :class:`scrapy.utils.datatypes.LocalCache` no longer evicts the oldest item
    when updating an existing one.
    (:gh:`8113`)

Documentation
~~~~~~~~~~~~~

-   Added a page :ref:`about using Scrapy with coding agents <agents>`,
    covering the official agent plugin and the Scrapy MCP server.
    (:gh:`8120`)

-   Documented the request metadata keys that were missing from the
    :ref:`list of special keys <topics-request-meta>`, and documented that keys
    whose name starts with an underscore are internal.
    (:gh:`3585`, :gh:`5564`, :gh:`7933`)

-   Documented :ref:`how to write custom Scrapy commands <custom-commands>`,
    :ref:`how to write custom spider templates <spider-templates>`, :ref:`how
    to access the response of a failed media download
    <media-pipeline-failed-downloads>` and :ref:`how to set request headers
    for media pipeline requests <media-request-headers>`.
    (:gh:`2046`,
    :gh:`2504`,
    :gh:`3056`,
    :gh:`6844`,
    :gh:`6904`,
    :gh:`8049`,
    :gh:`8056`,
    :gh:`8063`)

-   Documented how to subclass
    :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` to
    allow or deny redirects based on the target URL, and how to set the
    priority of the requests that :class:`~scrapy.spiders.CrawlSpider`
    generates from its rules.
    (:gh:`3613`, :gh:`4009`, :gh:`8061`, :gh:`8070`)

-   Documented that :func:`~scrapy.utils.project.get_project_settings`
    returns project settings only, that spiders :ref:`running concurrently in
    the same process <run-multiple-spiders>` get independent crawlers,
    middlewares and settings, and that Scrapy sets the ``request`` attribute
    of the :class:`~twisted.python.failure.Failure` objects that it passes to
    errbacks.
    (:gh:`2378`,
    :gh:`4253`,
    :gh:`6408`,
    :gh:`7901`,
    :gh:`8051`,
    :gh:`8055`)

-   Documented :class:`~scrapy.commands.ScrapyCommand` and
    :class:`~scrapy.statscollectors.StatsCollector`.
    (:gh:`6844`, :gh:`6904`, :gh:`8072`)

-   Improved the contents of ``llms-full.txt`` and Markdown versions of
    documentation pages.
    (:gh:`8083`,
    :gh:`8087`,
    :gh:`8088`,
    :gh:`8089`,
    :gh:`8117`,
    :gh:`8119`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Reached 100% test coverage.
    (:gh:`8021`)

-   Improved and fixed type hints.
    (:gh:`8017`, :gh:`8077`, :gh:`8102`)

-   Improved CodSpeed benchmarks.
    (:gh:`8030`, :gh:`8080`)

-   CI and test improvements and fixes.
    (:gh:`8027`,
    :gh:`8039`,
    :gh:`8059`,
    :gh:`8064`,
    :gh:`8073`,
    :gh:`8075`,
    :gh:`8076`,
    :gh:`8079`,
    :gh:`8092`,
    :gh:`8103`,
    :gh:`8115`)

.. _release-2.18.0:

Scrapy 2.18.0 (2026-08-20)
--------------------------

Highlights:

-   ``HttpxDownloadHandler`` now uses `httpx2 <https://httpx2.pydantic.dev/>`__

-   The Twisted-based HTTP/2 download handler is no longer experimental

-   ``brotli`` and Zstandard support are now always available, and :ref:`optional
    extras <extras>` cover the rest of the optional features

-   Late :class:`~scrapy.crawler.Crawler` attributes, such as
    :attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError`
    instead of being ``None`` before the crawl starts

-   Item exporters now export fields in declaration order

-   New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`

-   New :ref:`optimization <optimize>` page and :ref:`built-in stats reference
    <topics-stats-reference>`

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   ``brotli`` (``brotlicffi`` on PyPy) and Zstandard support (the standard
    library :mod:`compression.zstd` module on Python 3.14 and higher, the
    ``backports.zstd`` package on earlier versions) are now required, so
    ``br`` and ``zstd`` are always included in the ``Accept-Encoding`` header
    of requests, and Brotli- and Zstandard-compressed responses are always
    decoded. Websites may now serve such responses to crawls that previously
    did not advertise support for them.

    The minimum required versions are ``brotli`` 1.2.0, ``brotlicffi``
    1.2.0.0 and ``backports.zstd`` 1.3.0.

    (:gh:`4698`, :gh:`6978`, :gh:`7083`, :gh:`7929`, :gh:`8009`)

-   Increased the minimum versions of the following dependencies:

    - cryptography_: 37.0.0 → 41.0.5

    - pyOpenSSL_: 22.0.0 → 24.3.0

    - queuelib_: 1.4.2 → 1.6.1

    - service_identity_: 23.1.0 → 24.2.0

    - w3lib_: 1.17.0 → 2.1.1

    (:gh:`7841`, :gh:`7874`, :gh:`7879`, :gh:`8001`)

-   The IPython :ref:`shell <topics-shell>` requires IPython 8.15.0 or higher.
    Install the :ref:`ipython extra <extras>` to get a compatible version.
    (:gh:`5447`, :gh:`7596`, :gh:`7816`)

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   The following runtime usage of zope.interface_ interfaces is removed:

    - :class:`~scrapy.spiderloader.SpiderLoader` and
      :class:`~scrapy.spiderloader.DummySpiderLoader` are no longer marked
      as implementing the ``ISpiderLoader`` interface.

    - :func:`~scrapy.spiderloader.get_spider_loader` no longer checks that the
      configured spider loader implements the ``ISpiderLoader`` interface.

    - :class:`~scrapy.extensions.feedexport.BlockingFeedStorage`,
      :class:`~scrapy.extensions.feedexport.FileFeedStorage` and
      :class:`~scrapy.extensions.feedexport.StdoutFeedStorage` are no longer
      marked as implementing the ``IFeedStorage`` interface.

    - :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no
      longer checks that the ``DOWNLOADER_CLIENTCONTEXTFACTORY`` class
      implements the ``IPolicyForHTTPS`` interface.

    (:gh:`6585`, :gh:`7731`)

-   The :attr:`~scrapy.crawler.Crawler.engine`,
    :attr:`~scrapy.crawler.Crawler.extensions`,
    :attr:`~scrapy.crawler.Crawler.logformatter`,
    :attr:`~scrapy.crawler.Crawler.request_fingerprinter` and
    :attr:`~scrapy.crawler.Crawler.stats` attributes of
    :class:`~scrapy.crawler.Crawler` raise :exc:`RuntimeError` when read before
    the crawl starts, instead of being ``None`` until then.

    Code that reads them from the :signal:`spider_opened` signal handler
    onwards is unaffected, and no longer needs to narrow their type. Code that
    checked whether they were set, e.g. ``if crawler.stats:``, must be updated,
    since reading them now raises instead of returning ``None``.

    (:gh:`6136`, :gh:`7882`)

-   :ref:`Item exporters <topics-exporters>` now export the fields of an item
    in declaration order, i.e. the order in which they are defined in the
    :ref:`item class <item-types>`, instead of the order in which they were
    populated, as :class:`~scrapy.exporters.CsvItemExporter` already did.
    :class:`dict` items, which have no declared fields, keep using the key
    order of each item.
    (:gh:`6662`, :gh:`6854`, :gh:`7824`)

-   ``scrapy.utils.serialize.ScrapyJSONEncoder``, used by :ref:`JSON feed
    exports <topics-feed-format-json>`, the :ref:`telnet console
    <topics-telnetconsole>` and the
    :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension, now
    serializes :class:`~datetime.datetime`, :class:`~datetime.date` and
    :class:`~datetime.time` objects in ISO 8601 format, e.g.
    ``2023-08-03T23:24:57.148903+00:00`` instead of ``2023-08-03 23:24:57``,
    keeping microseconds and time zone information.

    Its ``DATE_FORMAT`` and ``TIME_FORMAT`` attributes are removed.

    (:gh:`2087`, :gh:`7918`)

-   ``scrapy.utils.trackref.live_refs`` is now a
    :class:`~weakref.WeakKeyDictionary` instead of a
    :class:`collections.defaultdict`, so that classes defined at run time are
    released once they are no longer used. Reading the entry of a class with no
    tracked instances now raises :exc:`KeyError` instead of creating and
    returning an empty mapping.
    (:gh:`5995`, :gh:`7922`)

-   The ``MEMDEBUG_NOTIFY`` setting is removed. It had no effect, but code
    reading it now gets ``None`` instead of its default value, which was an
    empty list.
    (:gh:`7737`)

-   ``scrapy.utils.log.logformatter_adapter()`` no longer passes the whole
    :class:`dict` returned by a :ref:`log formatter <custom-log-formats>`
    method as logging arguments when that ``dict`` has no ``args`` key, or its
    ``args`` are empty, and its ``msg`` has no ``%(name)s`` placeholders. Such
    messages are now logged verbatim, so a literal ``%`` in them no longer
    breaks logging.

    An ``args`` :class:`tuple` is now expanded into one logging argument per
    item, so that ``%``-style placeholders work with it as they do with a
    ``dict``.

    (:gh:`5570`, :gh:`5572`, :gh:`7936`)

-   :setting:`FEEDS` keys and ``FEED_URI`` values that are
    :class:`pathlib.Path` objects are now used as paths, instead of being
    converted into ``file://`` URIs. This makes them keep working when they
    contain :ref:`URI parameters <topics-feed-uri-params>` or characters that
    URI conversion would percent-encode.
    (:gh:`5794`, :gh:`6425`, :gh:`6611`, :gh:`7674`)

-   :class:`~scrapy.Selector` and :attr:`TextResponse.selector
    <scrapy.http.TextResponse.selector>` no longer force the ``html`` selector
    type for responses that are neither :class:`~scrapy.http.HtmlResponse` nor
    :class:`~scrapy.http.XmlResponse` objects. A
    :class:`~scrapy.http.JsonResponse` gets the ``json`` type, and for any
    other response ``parsel`` determines the type from the body.

    The response class, and hence the selector type, comes from the content
    type that the website reports. When a website reports the wrong content
    type, recast the response, e.g. ``response.replace(cls=HtmlResponse)``.

    (:gh:`4627`, :gh:`5291`, :gh:`6025`, :gh:`7924`, :gh:`7972`)

-   :ref:`AutoThrottle <topics-autothrottle>` no longer sets the
    ``download_delay`` attribute of the running spider to define the starting
    delay of download slots. The starting delay is still applied, but code
    that reads that attribute at run time no longer sees it.
    (:gh:`7167`, :gh:`7175`, :gh:`7833`)

-   The :command:`check` command now ignores :setting:`ITEM_PIPELINES` and
    :setting:`FEEDS`, since contracts check the output of callbacks instead of
    sending it to item processing, so a check run no longer triggers their side
    effects, e.g. writing an empty output file. Use the ``-s`` command-line
    option to set them back for a check run.
    (:gh:`3385`, :gh:`7957`)

-   :class:`~scrapy.spiders.XMLFeedSpider` and
    :class:`~scrapy.spiders.CSVFeedSpider` no longer raise
    :exc:`~scrapy.exceptions.NotConfigured` when ``parse_node()`` or
    ``parse_row()`` is not defined; the resulting :exc:`AttributeError` is
    reported instead.
    (:gh:`7768`)

Deprecation removals
~~~~~~~~~~~~~~~~~~~~

-   ``scrapy.utils.misc.md5sum()``, deprecated since Scrapy 2.12.0, is
    removed.
    (:gh:`6264`, :gh:`8023`)

-   ``scrapy.utils.iterators.xmliter()``, deprecated since Scrapy 2.11.1
    because it is vulnerable to ReDoS attacks, is removed. Use
    :func:`~scrapy.utils.iterators.xmliter_lxml` instead.
    (:gh:`7765`)

-   ``scrapy.utils.datatypes.CaselessDict``, deprecated since Scrapy 2.10.0,
    is removed. Use
    :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` instead.
    (:gh:`5146`, :gh:`8023`)

Deprecations
~~~~~~~~~~~~

-   The ``download_delay`` spider attribute is deprecated. Use the
    :setting:`DOWNLOAD_DELAY` setting, or :setting:`DOWNLOAD_SLOTS` to set a
    delay for specific domains, instead.

    The ``max_concurrent_requests`` spider attribute, deprecated since Scrapy
    2.14.0, now sets the :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` setting,
    which is what it always mapped to, and warns accordingly.

    Both attributes are ignored, with a different warning, when the
    corresponding setting is already set at the ``spider`` priority or higher.

    (:gh:`7167`, :gh:`7175`, :gh:`7833`)

-   The ``Spider.log()`` method is deprecated. Use the methods of
    :attr:`Spider.logger <scrapy.Spider.logger>` instead.
    (:gh:`7739`)

-   The ``scrapy.interfaces`` module and its ``ISpiderLoader`` interface are
    deprecated. Custom spider loaders only need to follow
    :class:`~scrapy.spiderloader.SpiderLoaderProtocol`.
    (:gh:`6585`, :gh:`7731`)

-   ``scrapy.extensions.feedexport.IFeedStorage`` is deprecated. Custom feed
    storages only need to follow
    ``scrapy.extensions.feedexport.FeedStorageProtocol``.
    (:gh:`6585`, :gh:`7731`)

-   ``scrapy.utils.python.re_rsearch()`` is deprecated.
    (:gh:`7765`)

-   Importing ``FileException`` from ``scrapy.pipelines.files`` is deprecated.
    Import it from ``scrapy.pipelines.media`` instead.
    (:gh:`7544`, :gh:`7673`, :gh:`7973`)

-   Setting ``request.meta["is_secure"]`` to ``False`` to send an ``s3://``
    request over plaintext HTTP is deprecated. The flag will be ignored in a
    future Scrapy version.
    (:gh:`7738`)

-   The unused ``multiplier`` attribute of
    :class:`~scrapy.extensions.periodic_log.PeriodicLog` is deprecated.
    (:gh:`7809`, :gh:`7982`)

-   Returning, from a :ref:`log formatter <custom-log-formats>` method, a
    ``msg`` with ``%(name)s`` placeholders and no ``args`` is deprecated. Those
    placeholders are still interpolated with the returned :class:`dict`, but in
    a future Scrapy version the message will be logged verbatim. Return those
    values under ``args`` instead.
    (:gh:`5570`, :gh:`7971`)

New features
~~~~~~~~~~~~

-   Added :ref:`optional extras <extras>` for every optional dependency of
    Scrapy: ``bpython``, ``gcs``, ``httpx``, ``images``, ``ipython``,
    ``ptpython``, ``robotparser``, ``s3``, ``twisted-http2`` and ``uvloop``.
    For example, ``pip install scrapy[s3,images]``.
    (:gh:`7596`)

-   :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` now
    uses `httpx2 <https://httpx2.pydantic.dev/>`__, the successor of ``httpx``,
    which the new :ref:`httpx extra <extras>` installs together with its HTTP/2
    and SOCKS proxy support. ``httpx`` is still used when ``httpx2`` is not
    installed, but it is no longer tested.
    (:gh:`7762`)

-   Added a :signal:`robots_parsed` signal, sent by
    :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` after
    it parses a :file:`robots.txt` file. It supports :ref:`asynchronous
    handlers <signal-deferred>`.

    Added a :meth:`~scrapy.robotstxt.RobotParser.crawl_delay` method to
    :class:`~scrapy.robotstxt.RobotParser`, implemented by all built-in
    :ref:`robots.txt parsers <topics-dlmw-robots>`.

    (:gh:`7830`)

-   Added a :meth:`Request.to_curl() <scrapy.Request.to_curl>` method, the
    inverse of :meth:`~scrapy.Request.from_curl`.
    (:gh:`7743`, :gh:`7746`, :gh:`7802`)

-   Added a :reqmeta:`depth_reset` request meta key that gives a request depth
    0 instead of the depth of its source response plus 1.
    (:gh:`891`, :gh:`7913`)

-   Added
    :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`,
    enabled by default, which warns once per crawl when a spider yields a
    request carrying internal :attr:`~scrapy.Request.meta` keys that were
    likely copied from ``response.meta``, and a
    :setting:`META_COPY_WARN_SKIP_KEYS` setting to exclude keys from that
    check.
    (:gh:`7588`)

-   Added an :setting:`AWS_MAX_POOL_CONNECTIONS` setting, which defines the
    connection pool size of the AWS clients of the :ref:`S3 feed storage
    backend <topics-feed-storage-s3>` and the :ref:`S3 media pipeline storage
    backend <media-pipelines-s3>`, and defaults to
    :setting:`REACTOR_THREADPOOL_MAXSIZE`. It is also exposed as a
    ``max_pool_connections`` parameter of ``S3FeedStorage`` and as an
    ``AWS_MAX_POOL_CONNECTIONS`` attribute of ``S3FilesStore``.
    (:gh:`4985`, :gh:`7794`)

-   Added a :func:`scrapy.utils.asyncio.sleep` function, which works both with
    and without a Twisted reactor.
    (:gh:`7843`)

-   :setting:`CONCURRENT_REQUESTS` can now be set to ``0`` for no limit.
    (:gh:`7840`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` is no
    longer experimental, and it now sends the :signal:`bytes_received` and
    :signal:`headers_received` signals and supports
    :exc:`~scrapy.exceptions.StopDownload`.
    (:gh:`5046`, :gh:`5047`, :gh:`5055`, :gh:`7896`, :gh:`7986`)

-   An exception raised by :meth:`Spider.start() <scrapy.Spider.start>` is now
    reported through the :signal:`spider_error` signal and the
    :stat:`spider_exceptions/count` and :stat:`spider_exceptions/{exception}`
    stats, and closes the spider with the new ``start_error``
    :stat:`finish_reason` instead of ``finished``. See :ref:`start-error`.

    :exc:`~scrapy.exceptions.CloseSpider` raised from :meth:`Spider.start()
    <scrapy.Spider.start>` now closes the spider with the given reason, instead
    of being reported as a start error.

    (:gh:`3463`, :gh:`4058`, :gh:`4182`, :gh:`6148`, :gh:`7884`)

-   :exc:`~scrapy.exceptions.CloseSpider` can now also be raised while the
    spider is starting, e.g. from a :signal:`spider_opened` signal handler or
    from the ``open_spider()`` method of an :ref:`item pipeline
    <topics-item-pipeline>`, to close the spider before it starts crawling.
    Every component still gets started, and stopped, before the spider is
    closed with the given reason.
    (:gh:`3435`, :gh:`7905`)

-   Added an :ref:`FTPS feed storage backend <feed-storage-ftps>`, i.e. support
    for the ``ftps`` URI scheme in :setting:`FEEDS`, which uploads the feed
    over a TLS connection, verifying the certificate of the server.
    (:gh:`4180`, :gh:`7953`)

-   Changes to :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>`
    during a crawl are now taken into account by
    :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, whose
    :meth:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware.should_follow`
    method is now documented as the way to implement a different offsite
    policy.
    (:gh:`3257`, :gh:`3412`, :gh:`7903`, :gh:`7912`)

-   :class:`~scrapy.settings.BaseSettings` methods that take settings, such as
    :meth:`~scrapy.settings.BaseSettings.update` and the ``settings`` parameter
    of crawler classes, now also accept an iterable of ``(name, value)``
    tuples.
    (:gh:`7759`, :gh:`7763`)

-   The ``cookies`` parameter of :class:`~scrapy.Request` now also accepts
    :class:`bool`, :class:`float` and :class:`int` values, and the ``formdata``
    parameter of :class:`~scrapy.FormRequest` now accepts any mapping or
    iterable of key-value pairs.
    (:gh:`7858`, :gh:`7864`)

-   Added a ``scrapy.utils.reactorless.uninstall_reactor_import_hook()``
    function, which :meth:`AsyncCrawlerProcess.start()
    <scrapy.crawler.AsyncCrawlerProcess.start>` now uses to uninstall the
    :mod:`twisted.internet.reactor` import hook when it exits.
    (:gh:`7747`)

-   Added the :stat:`depth/request_ignored_count` and
    :stat:`httpcache/retrieve_error` stats.
    (:gh:`1308`, :gh:`2222`, :gh:`7805`, :gh:`7916`)

-   The :meth:`~scrapy.exporters.BaseItemExporter.get_serialized_fields` method
    of :ref:`item exporters <topics-exporters>`, previously named
    ``_get_serialized_fields()``, is now public and documented, for
    :ref:`custom item exporters <custom-exporters>` to use.
    (:gh:`5706`, :gh:`7931`)

-   Scrapy now writes the session keys of its HTTPS connections to the file
    that the ``SSLKEYLOGFILE`` environment variable points to, so that traffic
    analysis tools such as Wireshark can decrypt them. See :ref:`debug-tls`.
    (:gh:`4368`, :gh:`7948`)

-   Added an :setting:`HTTP2_MAX_FRAME_SIZE` setting, which allows raising the
    maximum HTTP/2 frame size that servers may send, previously fixed at
    16384, above which connections failed.
    (:gh:`5050`, :gh:`7988`)

-   The :command:`crawl`, :command:`parse` and :command:`runspider` commands
    now warn when :setting:`FEEDS` is set, e.g. through ``-o`` or ``-O``, but
    the :class:`~scrapy.extensions.feedexport.FeedExporter` extension is
    disabled, so that no item is exported.
    (:gh:`5970`, :gh:`6082`, :gh:`6373`, :gh:`7902`)

-   Log formatters (:setting:`LOG_FORMATTER`), item processors
    (:setting:`ITEM_PROCESSOR`) and :ref:`robots.txt parsers
    <topics-dlmw-robots>` (:setting:`ROBOTSTXT_PARSER`) are now built as
    :ref:`components <topics-components>`, so they no longer need a
    ``from_crawler()`` method.
    (:gh:`7808`)

Bug fixes
~~~~~~~~~

-   :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now
    logs a warning and handles the request as a cache miss when reading a cache
    entry raises an exception, e.g. because the entry is corrupted, instead of
    letting the exception propagate. It also counts those entries in the new
    :stat:`httpcache/retrieve_error` stat.
    (:gh:`2222`, :gh:`7805`)

-   :ref:`Feed URIs <topics-feed-uri-params>` now only expand ``%(...)s``
    parameters, keeping any other percent character as is, so that
    percent-encoded URIs, e.g. one with ``%20`` in a path or with
    percent-encoded FTP credentials, are no longer misinterpreted as
    printf-style formatting directives.
    (:gh:`5794`, :gh:`6425`, :gh:`7674`)

-   :ref:`Feed exports <topics-feed-exports>` now start storing a
    :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` batch as soon as it is complete,
    instead of waiting until the spider closes.
    (:gh:`7730`, :gh:`7733`)

-   :class:`~scrapy.exporters.CsvItemExporter` now warns when the fields that
    it took from the first item do not cover the fields of a later item, i.e.
    when it silently drops data.
    (:gh:`4002`, :gh:`4053`, :gh:`7613`, :gh:`7651`)

-   ``GCSFeedStorage`` no longer requires the ``storage.buckets.get``
    permission.
    (:gh:`5475`, :gh:`7945`)

-   :ref:`Media pipelines <topics-media-pipeline>` now log media requests that
    were filtered out, e.g. as offsite requests, at the ``DEBUG`` level and
    without a traceback, instead of reporting them as download errors.
    (:gh:`7544`, :gh:`7673`)

-   :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` now raises
    :exc:`~scrapy.exceptions.IgnoreRequest` with a message, e.g. ``Filtered
    offsite request to 'offsite.example'``, which errbacks and log messages
    that report that exception now include.
    (:gh:`7544`, :gh:`7673`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    skips response header lines that have no colon, logging them at the
    ``DEBUG`` level, as web browsers do, instead of being unable to download
    such a response at all.
    (:gh:`210`, :gh:`7806`)

-   :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` now sends
    domain cookies to hosts without a dot in their name and to hosts given as
    an IP address.
    (:gh:`6410`, :gh:`7900`)

-   :meth:`TextResponse.json() <scrapy.http.TextResponse.json>` now decodes
    bodies that are not valid UTF-8, UTF-16 or UTF-32 using
    :attr:`TextResponse.encoding <scrapy.http.TextResponse.encoding>`, instead
    of raising :exc:`UnicodeDecodeError`.
    (:gh:`6456`, :gh:`7897`)

-   ``scrapy.resolver.CachingHostnameResolver`` now caches addresses without a
    port, and sets the requested port on cache hits, so that a cached address
    no longer carries the port of the request that populated the cache.
    (:gh:`6442`, :gh:`7772`)

-   :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` now removes the
    directory of a download slot from the :setting:`JOBDIR` directory once that
    slot is drained.
    (:gh:`5275`, :gh:`7955`)

-   :class:`~scrapy.extensions.telnet.TelnetConsole` no longer raises an
    exception on shutdown when it could not listen on any of the
    :setting:`TELNETCONSOLE_PORT` ports.
    (:gh:`2702`, :gh:`7910`)

-   The :setting:`DOWNLOAD_WARNSIZE` warning is no longer logged twice for a
    response whose ``Content-Length`` header already exceeded the limit.
    (:gh:`2476`, :gh:`7963`)

-   :class:`HttpCompressionMiddleware
    <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`
    now logs a warning when it drops a response for exceeding
    :setting:`DOWNLOAD_MAXSIZE` during decompression.
    (:gh:`6616`, :gh:`7742`)

-   :class:`~scrapy.spidermiddlewares.depth.DepthMiddleware` now logs only the
    first request ignored for exceeding :setting:`DEPTH_LIMIT`, and counts them
    all in the new :stat:`depth/request_ignored_count` stat.
    (:gh:`1308`, :gh:`7916`)

-   :command:`parse` now sets the callback it uses on the request of the
    response it passes to that callback.
    (:gh:`3095`, :gh:`3124`, :gh:`7803`)

-   The IPython :ref:`shell <topics-shell>` now works when an asyncio event
    loop is already running in the same thread, e.g. when calling
    ``scrapy.shell.inspect_response()`` from a callback while using the asyncio
    reactor.
    (:gh:`5447`, :gh:`7816`)

-   :meth:`Request.from_curl() <scrapy.Request.from_curl>` now merges repeated
    ``-d``, ``--data`` and ``--data-raw`` options into a single body joined
    with ``&``, as curl does, instead of keeping only the last one.
    (:gh:`7728`)

-   The ``copy()`` method and the ``|=`` operator of
    ``scrapy.utils.datatypes.CaseInsensitiveDict`` no longer leave the internal
    mapping of original key spellings shared or out of date.
    (:gh:`7783`)

-   :meth:`ExecutionEngine.download_async()
    <scrapy.core.engine.ExecutionEngine.download_async>` no longer recurses
    once per returned request, e.g. once per redirect.
    (:gh:`7544`, :gh:`7673`)

-   :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
    now canonicalizes each extracted URL once instead of twice when
    ``canonicalize`` is ``True``.
    (:gh:`7961`)

-   Items yielded from :meth:`Spider.start() <scrapy.Spider.start>` now keep
    the spider busy until the :ref:`item pipelines <topics-item-pipeline>` are
    done with them, so that a spider that only yields items from
    :meth:`~scrapy.Spider.start` no longer closes before processing them.
    (:gh:`7029`, :gh:`7891`)

-   :func:`~scrapy.utils.response.open_in_browser` now also adds its ``base``
    tag to HTML responses that have no ``head`` element, and it now overrides a
    ``base`` tag already present in the response, so that relative URLs resolve
    against the response URL in every case.
    (:gh:`6550`, :gh:`7879`)

-   :meth:`Spider.start() <scrapy.Spider.start>` implementations that are not
    asynchronous generators now raise :exc:`TypeError` with a message that says
    so, instead of failing in a way that does not point at the cause.
    (:gh:`5426`, :gh:`7946`)

-   The :command:`check`, :command:`fetch` and :command:`parse` commands now
    return the exit code 1 when a component fails to initialize, as
    :command:`crawl` and :command:`runspider` already did.
    (:gh:`4292`, :gh:`7920`)

-   :class:`HttpCompressionMiddleware
    <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>`
    no longer hangs on a ``deflate`` response body followed by extra bytes.
    (:gh:`7841`)

-   :func:`scrapy.utils.python.get_func_args` now reports the parameters that a
    :class:`functools.partial` object binds by position, instead of an empty
    list.
    (:gh:`7841`)

-   Fixed :exc:`NameError` exceptions on Python 3.14, where :pep:`649` made
    annotation evaluation lazy, when inspecting the signature of a callable
    with annotations imported only for type checking.
    (:gh:`7796`, :gh:`7818`)

-   ``scrapy.utils.decorators.deprecated`` can now be used both as
    ``@deprecated`` and as ``@deprecated(...)`` without confusing type
    checkers.
    (:gh:`7797`)

-   The default download handlers can now download from domains with emoji
    characters or underscores, which were previously rejected.
    (:gh:`3321`, :gh:`4330`, :gh:`7846`)

-   Callbacks and media pipeline results no longer wait 100 ms before
    proceeding.
    (:gh:`8019`)

-   Shutting down a crawl no longer risks raising an unhandled
    :exc:`RuntimeError` if the code interrupted by the shutdown signal was
    itself writing to the log.
    (:gh:`8022`)

-   Nested selectors, e.g. the result of calling
    :meth:`~scrapy.Selector.jmespath` on a selector, now let ``parsel``
    determine their type instead of forcing the ``html`` type, so that they no
    longer return the wrong type or value.
    (:gh:`8038`, :gh:`8040`)

Documentation
~~~~~~~~~~~~~

-   Added a :ref:`built-in stats reference <topics-stats-reference>`, covering
    every stat that Scrapy sets.
    (:gh:`6351`, :gh:`7814`)

-   Replaced the broad crawls page with a new :ref:`optimization <optimize>`
    page, about finding the bottleneck of a crawl before changing any setting,
    which covers :ref:`broad crawls <broad-crawls>` as one of its sections.
    (:gh:`4737`, :gh:`7938`)

-   Added a :ref:`concepts <concepts>` page, a quick map of Scrapy's main
    building blocks, what problem each one solves, and when to reach for it.
    (:gh:`1569`, :gh:`8025`)

-   Added a :ref:`cookies <cookies>` page, which gathers what used to be
    spread across the request and downloader middleware pages.
    (:gh:`7947`)

-   Added :ref:`callbacks <callbacks>` and :ref:`errbacks <errbacks>` sections
    to the request and response page, covering :ref:`callback assignment
    <callback-assignment>`, :ref:`how to write a callback <writing-callbacks>`
    and :ref:`supported callback output <callback-output>`.
    (:gh:`5054`, :gh:`6437`, :gh:`7821`, :gh:`7898`)

-   Documented the :setting:`ITEM_PROCESSOR` setting and the
    :class:`~scrapy.pipelines.ItemProcessorProtocol` protocol that its value
    must implement.
    (:gh:`7983`)

-   Documented :ref:`how to write an item exporter <custom-exporters>`,
    :ref:`how to test an item pipeline <test-item-pipeline>`, :ref:`how to
    download a request from a downloader middleware <mw-download>`, :ref:`how
    to name media files after the response <file-naming-response>`, :ref:`how
    to add objects to the shell <shell-update-vars>` and :ref:`how to run
    spiders inside an existing application <run-spiders-in-apps>` or :ref:`in a
    Jupyter notebook <run-in-notebook>`.
    (:gh:`915`,
    :gh:`1199`,
    :gh:`2594`,
    :gh:`5706`,
    :gh:`6554`,
    :gh:`6594`,
    :gh:`7751`,
    :gh:`7872`,
    :gh:`7876`,
    :gh:`7889`,
    :gh:`7909`,
    :gh:`7931`)

-   Added an :ref:`inspecting live traffic <debug-live-traffic>` section to the
    debugging page, covering Wireshark and mitmproxy.
    (:gh:`5222`, :gh:`8007`)

-   Documented how :class:`~scrapy.http.TextResponse` resolves the response
    encoding, and how to resolve it differently, e.g. to give the encoding
    declared in the response body precedence over the ``Content-Type`` header.
    (:gh:`4933`, :gh:`7977`)

-   Documented the :ref:`memory use of response parsing
    <security-response-size>` and the :ref:`parser limits
    <security-parser-limits>` that Scrapy lifts, in the security page.
    (:gh:`5700`, :gh:`7930`)

-   Documented that :ref:`signal handlers run in an undefined order
    <signal-order>`, that :signal:`scheduler_empty` must only be awaited from
    :meth:`~scrapy.Spider.start`, that concurrency and politeness settings
    apply per crawler when :ref:`running multiple spiders in the same process
    <run-multiple-spiders>`, and that a :setting:`JOBDIR` directory cannot be
    shared across Scrapy versions.
    (:gh:`3191`,
    :gh:`5330`,
    :gh:`5522`,
    :gh:`7861`,
    :gh:`7883`,
    :gh:`7907`,
    :gh:`7941`)

-   Documented that the ``html`` iterator of
    :class:`~scrapy.spiders.XMLFeedSpider` can silently mangle tags that HTML
    treats as void elements, e.g. ``<link>``, dropping their content and
    closing tag.
    (:gh:`4675`, :gh:`8045`)

-   Documented that the ``keep_fragments`` parameter of
    :func:`~scrapy.utils.request.fingerprint` is not a substitute for
    rendering JavaScript to reach content that a headless browser loads based
    on the URL fragment.
    (:gh:`4789`, :gh:`8033`)

-   Documented :ref:`how to derive the job directory from the spider name
    <job-dir-spider-name>`.
    (:gh:`4748`, :gh:`8035`)

-   Documented that the project name from :file:`scrapy.cfg` also appears
    elsewhere by default, and which of those uses actually require it to
    match.
    (:gh:`2484`, :gh:`8044`)

-   Many other corrections and improvements.
    (:gh:`4589`,
    :gh:`4796`,
    :gh:`5532`,
    :gh:`5548`,
    :gh:`6053`,
    :gh:`6184`,
    :gh:`6627`,
    :gh:`6787`,
    :gh:`6943`,
    :gh:`6989`,
    :gh:`7710`,
    :gh:`7725`,
    :gh:`7737`,
    :gh:`7767`,
    :gh:`7769`,
    :gh:`7771`,
    :gh:`7774`,
    :gh:`7775`,
    :gh:`7777`,
    :gh:`7779`,
    :gh:`7780`,
    :gh:`7817`,
    :gh:`7832`,
    :gh:`7835`,
    :gh:`7862`,
    :gh:`7871`,
    :gh:`7875`,
    :gh:`7880`,
    :gh:`7890`,
    :gh:`7903`,
    :gh:`7913`,
    :gh:`7917`,
    :gh:`7939`,
    :gh:`7940`,
    :gh:`7962`,
    :gh:`7965`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Improved and fixed type hints.
    (:gh:`7712`,
    :gh:`7785`,
    :gh:`7858`,
    :gh:`7864`,
    :gh:`7865`,
    :gh:`7867`)

-   Added CPU benchmarks, tracked on CodSpeed, so that performance regressions
    are caught before they are merged and performance work can be measured.
    (:gh:`7831`,
    :gh:`7839`,
    :gh:`7870`,
    :gh:`7887`,
    :gh:`7914`,
    :gh:`7954`)

-   Added a nightly job that runs the test suite against the development
    branches of dependencies, so that incompatibilities are found before those
    dependencies are released.
    (:gh:`5291`, :gh:`6025`, :gh:`7924`, :gh:`7960`)

-   CI and test improvements and fixes.
    (:gh:`5049`,
    :gh:`5620`,
    :gh:`5837`,
    :gh:`6478`,
    :gh:`6794`,
    :gh:`7262`,
    :gh:`7437`,
    :gh:`7702`,
    :gh:`7720`,
    :gh:`7724`,
    :gh:`7727`,
    :gh:`7736`,
    :gh:`7741`,
    :gh:`7749`,
    :gh:`7753`,
    :gh:`7755`,
    :gh:`7768`,
    :gh:`7778`,
    :gh:`7782`,
    :gh:`7792`,
    :gh:`7793`,
    :gh:`7795`,
    :gh:`7797`,
    :gh:`7798`,
    :gh:`7809`,
    :gh:`7829`,
    :gh:`7834`,
    :gh:`7836`,
    :gh:`7838`,
    :gh:`7841`,
    :gh:`7844`,
    :gh:`7848`,
    :gh:`7853`,
    :gh:`7854`,
    :gh:`7857`,
    :gh:`7859`,
    :gh:`7863`,
    :gh:`7895`,
    :gh:`7906`,
    :gh:`7928`,
    :gh:`7935`,
    :gh:`7966`,
    :gh:`7968`,
    :gh:`7974`,
    :gh:`7979`,
    :gh:`7985`,
    :gh:`7990`,
    :gh:`7993`,
    :gh:`7995`,
    :gh:`8000`,
    :gh:`8001`,
    :gh:`8002`)

.. _release-2.17.0:

Scrapy 2.17.0 (2026-07-07)
--------------------------

Highlights:

-   Security bug fixes

-   HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler``

-   Improved settings for changing allowed TLS versions

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP.

    Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler`
    sent signed S3 requests over plaintext HTTP unless
    ``request.meta["is_secure"]`` was set to a true value, exposing the request
    path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header
    (when using temporary credentials), and the response contents to network
    attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_
    security advisory for details.

    To restore the previous behavior for a given request, set
    ``request.meta["is_secure"]`` to ``False``.

    .. _76g3-c3x4-crvx: https://github.com/scrapy/scrapy/security/advisories/GHSA-76g3-c3x4-crvx

Deprecations
~~~~~~~~~~~~

-   The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use
    the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or
    :setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change
    the TLS method selection.
    (:gh:`3288`, :gh:`6546`)

-   The following spider attributes are deprecated in favor of settings:

    - ``http_user`` (use :setting:`HTTPAUTH_USER`)

    - ``http_pass`` (use :setting:`HTTPAUTH_PASS`)

    - ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`)

    (:gh:`7590`)

-   The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was
    never called by Scrapy.
    (:gh:`7626`, :gh:`7633`)

-   The following TLS-related functions and constants, intended for internal
    use, are deprecated:

    - ``scrapy.core.downloader.tls.METHOD_TLS``

    - ``scrapy.core.downloader.tls.METHOD_TLSv10``

    - ``scrapy.core.downloader.tls.METHOD_TLSv11``

    - ``scrapy.core.downloader.tls.METHOD_TLSv12``

    - ``scrapy.core.downloader.tls.openssl_methods``

    - ``scrapy.core.downloader.tls.DEFAULT_CIPHERS``

    - ``scrapy.utils.ssl.ffi_buf_to_string()``

    - ``scrapy.utils.ssl.get_temp_key_info()``

    - ``scrapy.utils.ssl.x509name_to_string()``

    (:gh:`6546`, :gh:`7619`, :gh:`7665`)

-   The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set
    ``follow=False`` in your rules to achieve the same effect.
    (:gh:`7592`)

-   Instantiating
    :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
    without a ``crawler`` argument is deprecated.
    (:gh:`7655`)

-   Instantiating
    :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a
    ``settings`` argument is deprecated.
    (:gh:`7664`)

New features
~~~~~~~~~~~~

-   Added support for HTTP/2 requests to
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It
    requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to
    ``True``.
    (:gh:`7575`)

-   Added support for SOCKS proxies to
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
    (:gh:`747`, :gh:`7575`)

-   Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and
    :setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the
    ``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated).
    Compared to the old setting, they support specifying a range of allowed
    versions and support newer TLS versions.
    (:gh:`4821`, :gh:`6546`)

-   Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and
    :setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`,
    :reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more
    flexible ways to set HTTP authentication data.
    (:gh:`7590`)

-   Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to
    skip request URL canonicalization.
    (:gh:`7473`)

-   Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor
    <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`.
    (:gh:`6321`, :gh:`7679`)

-   :attr:`scrapy.Item.fields` now returns the fields in the definition order
    instead of the alphabetical one.
    (:gh:`7015`, :gh:`7694`)

-   Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a
    :reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument
    of the
    :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that
    allow changing the log level of the message logged when the retry limit has
    been reached.
    (:gh:`4622`, :gh:`5297`, :gh:`7567`)

-   It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to
    ``None`` to use the default ciphers of the underlying TLS implementation.
    (:gh:`7499`, :gh:`7665`)

Improvements
~~~~~~~~~~~~

-   :class:`~scrapy.FormRequest` is no longer deprecated, only its
    ``from_response()`` method is still deprecated.
    (:gh:`7561`, :gh:`7671`)

-   Switched the item definition in the default project template from a
    :class:`scrapy.item.Item` to a dataclass.
    (:gh:`7493`, :gh:`7513`)

-   Fixed deprecation warnings with pyOpenSSL 26.3.0.
    (:gh:`7619`)

-   Removed the runtime warnings for :attr:`Spider.allowed_domains
    <scrapy.Spider.allowed_domains>` containing URLs or domains with ports
    instead of just domains and for spider classes having a ``start_url``
    attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use
    :doc:`scrapy-lint <scrapy-lint:index>` to find mistakes in your spider code
    instead.
    (:gh:`4421`, :gh:`7627`)

-   :func:`scrapy.utils.test.get_crawler` now disables
    :setting:`TELNETCONSOLE_ENABLED` by default.
    (:gh:`7644`)

-   Other code refactoring and improvements.
    (:gh:`7409`, :gh:`7593`, :gh:`7594`, :gh:`7611`, :gh:`7649`)

Bug fixes
~~~~~~~~~

-   :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no
    longer ignores proxy credentials for redirected or retried requests.
    (:gh:`7601`, :gh:`7630`)

-   :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the
    temporary file after the upload.
    (:gh:`7546`)

-   Fixed ``scrapy shell <URL>`` running a full spider crawl when there is a
    spider for the requested URL. This bug was introduced in Scrapy 2.13.0.
    (:gh:`7552`, :gh:`7557`)

-   The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL`
    settings are no longer ignored. This bug was introduced in Scrapy 2.12.0.
    (:gh:`7597`, :gh:`7614`)

-   :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes
    the connection after making the request.
    (:gh:`7602`, :gh:`7667`)

-   Removed the deprecated ``spider`` argument from the pipeline defined in the
    default project template.
    (:gh:`7676`)

-   Fixed ``scrapy genspider --edit`` not working.
    (:gh:`7260`, :gh:`7683`)

-   When a :class:`~scrapy.crawler.Crawler` instance is passed to
    :meth:`AsyncCrawlerRunner.create_crawler()
    <scrapy.crawler.AsyncCrawlerRunner.create_crawler>` or
    :meth:`CrawlerRunner.create_crawler()
    <scrapy.crawler.CrawlerRunner.create_crawler>`, settings from both classes
    are now merged, previously only the settings from the
    :class:`~scrapy.crawler.Crawler` instance were used.
    (:gh:`1280`, :gh:`7647`)

-   Fixed several issues with cookie handling in
    :func:`scrapy.utils.request.request_to_curl`.
    (:gh:`7603`, :gh:`7675`, :gh:`7684`)

-   Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the
    cache when :setting:`DNSCACHE_ENABLED` is set to ``False``.
    (:gh:`7663`)

-   Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments
    when looking for the ``<base>`` tag.
    (:gh:`7506`)

-   Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR`
    implementations.
    (:gh:`7589`)

-   Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with
    credentials.
    (:gh:`7604`, :gh:`7605`)

-   :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when
    looking for "nofollow" strings.
    (:gh:`7632`)

-   Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing
    some malformed sitemaps.
    (:gh:`7686`, :gh:`7687`)

Documentation
~~~~~~~~~~~~~

-   Mentioned :doc:`scrapy-lint <scrapy-lint:index>` in the docs.
    (:gh:`4421`, :gh:`7627`)

-   Added the docs about :ref:`security considerations <security>`.
    (:gh:`7389`, :gh:`7678`)

-   Improved the :ref:`item pipeline docs <topics-item-pipeline>`.
    (:gh:`2350`, :gh:`7676`)

-   Documented which stats are collected by
    :class:`~scrapy.extensions.corestats.CoreStats`.
    (:gh:`7421`)

-   Switched documentation examples from using :class:`scrapy.item.Item` to
    using dataclasses.
    (:gh:`7493`, :gh:`7513`)

-   Added feature comparison tables to the :ref:`download handler
    <download-handlers-ref>` docs.
    (:gh:`7575`)

-   Improved the docs for :ref:`logging settings <logging-settings>`.
    (:gh:`6909`, :gh:`7668`)

-   Documented a way to :ref:`improve startup time and memory usage
    <large-project-startup>` by using :setting:`SPIDER_MODULES`.
    (:gh:`7576`, :gh:`7600`)

-   Clarified handling of the ``type`` argument of :class:`~scrapy.Selector`.
    (:gh:`7704`)

-   Other documentation improvements and fixes.
    (:gh:`4954`,
    :gh:`6120`,
    :gh:`7286`,
    :gh:`7564`,
    :gh:`7573`,
    :gh:`7598`,
    :gh:`7599`,
    :gh:`7698`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Fixed deprecation warnings with pytest 9.1.0.
    (:gh:`7621`)

-   Type hints improvements and fixes.
    (:gh:`6958`, :gh:`7586`)

-   CI and test improvements and fixes.
    (:gh:`5954`,
    :gh:`7002`,
    :gh:`7017`,
    :gh:`7247`,
    :gh:`7508`,
    :gh:`7545`,
    :gh:`7566`,
    :gh:`7574`,
    :gh:`7585`,
    :gh:`7595`,
    :gh:`7608`,
    :gh:`7610`,
    :gh:`7612`,
    :gh:`7616`,
    :gh:`7625`,
    :gh:`7637`,
    :gh:`7639`,
    :gh:`7640`,
    :gh:`7641`,
    :gh:`7642`,
    :gh:`7643`,
    :gh:`7644`,
    :gh:`7645`,
    :gh:`7646`,
    :gh:`7654`,
    :gh:`7655`,
    :gh:`7664`,
    :gh:`7672`,
    :gh:`7677`,
    :gh:`7680`,
    :gh:`7682`,
    :gh:`7692`)

.. _release-2.16.0:

Scrapy 2.16.0 (2026-05-19)
--------------------------

Highlights:

-   Official support for Python 3.14

-   Support for Twisted 26.4.0+

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   Increased the minimum versions of the following dependencies:

    - service_identity_: 18.1.0 → 23.1.0

    (:gh:`7347`)

-   Added support for Twisted 26.4.0+.
    (:gh:`7347`, :gh:`7505`, :gh:`7520`)

-   Added support for Python 3.14.
    (:gh:`6604`, :gh:`7460`)

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   The following classes and functions, intended for internal use by
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    and :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`, have
    been made private:

    - ``scrapy.core.downloader.handlers.http11.ScrapyAgent``

    - ``scrapy.core.downloader.handlers.http11.ScrapyProxyAgent``

    - ``scrapy.core.downloader.handlers.http11.TunnelingAgent``

    - ``scrapy.core.downloader.handlers.http11.TunnelingTCP4ClientEndpoint``

    - ``scrapy.core.downloader.handlers.http11.tunnel_request_data()``

    - ``scrapy.core.downloader.handlers.http2.ScrapyH2Agent``

    (:gh:`7496`, :gh:`7510`)

Deprecations
~~~~~~~~~~~~

-   ``scrapy.FormRequest`` is deprecated. You can use the :doc:`form2request
    <form2request:index>` library instead, see :ref:`form`.
    (:gh:`6438`)

-   ``scrapy.utils.python.MutableChain`` is deprecated.
    (:gh:`7504`)

Deprecation removals
~~~~~~~~~~~~~~~~~~~~

-   The ``start_requests()`` method of :class:`~scrapy.Spider`, deprecated in
    2.13.0, is removed and no longer called. Use :meth:`~scrapy.Spider.start`
    instead, or both to maintain support for lower Scrapy versions.
    (:gh:`7490`)

-   Support for ``process_start_requests()`` methods of :ref:`spider middlewares
    <topics-spider-middleware>`, deprecated in 2.13.0, is removed. Use
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
    or both to maintain support for lower Scrapy versions.
    (:gh:`7490`)

-   Support for synchronous ``process_spider_output()`` methods of spider
    middlewares, deprecated in Scrapy 2.13.0, is removed. You should upgrade
    the affected middlewares to have asynchronous ``process_spider_output()``
    methods.
    (:gh:`7504`)

-   The ``spider`` arguments of the following methods of
    :class:`~scrapy.core.scraper.Scraper`, deprecated in Scrapy 2.13.0, are
    removed:

    - ``close_spider()``

    - ``enqueue_scrape()``

    - ``handle_spider_error()``

    - ``handle_spider_output()``

    (:gh:`7487`)

-   HTTP/1.0 support code, deprecated in Scrapy 2.13.0, is removed. This
    includes:

    - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``

    - The ``scrapy.core.downloader.webclient`` module.

    - The ``DOWNLOADER_HTTPCLIENTFACTORY`` setting.

    (:gh:`7486`)

-   The following functions, deprecated in Scrapy 2.13.0, are removed, you
    should import them from :mod:`w3lib.url` directly instead:

    - ``scrapy.utils.url.add_or_replace_parameter()``

    - ``scrapy.utils.url.add_or_replace_parameters()``

    - ``scrapy.utils.url.any_to_uri()``

    - ``scrapy.utils.url.canonicalize_url()``

    - ``scrapy.utils.url.file_uri_to_path()``

    - ``scrapy.utils.url.is_url()``

    - ``scrapy.utils.url.parse_data_uri()``

    - ``scrapy.utils.url.parse_url()``

    - ``scrapy.utils.url.path_to_file_uri()``

    - ``scrapy.utils.url.safe_download_url()``

    - ``scrapy.utils.url.safe_url_string()``

    - ``scrapy.utils.url.url_query_cleaner()``

    - ``scrapy.utils.url.url_query_parameter()``

    (:gh:`7487`)

-   The following test-related code, deprecated in Scrapy 2.13.0, is removed:

    - the ``scrapy.utils.testproc`` module

    - the ``scrapy.utils.testsite`` module

    - ``scrapy.utils.test.assert_gcs_environ()``

    - ``scrapy.utils.test.get_ftp_content_and_delete()``

    - ``scrapy.utils.test.get_gcs_content_and_delete()``

    - ``scrapy.utils.test.mock_google_cloud_storage()``

    - ``scrapy.utils.test.skip_if_no_boto()``

    - ``scrapy.utils.test.TestSpider``

    (:gh:`7487`)

-   ``scrapy.utils.versions.scrapy_components_versions()``, deprecated in
    Scrapy 2.13.0, is removed, you can use
    ``scrapy.utils.versions.get_versions()`` instead.
    (:gh:`7487`)

-   ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` and
    ``scrapy.utils.url.escape_ajax()``, deprecated in Scrapy 2.13.0, are
    removed.
    (:gh:`7487`)

-   The ``__init__()`` method of priority queue classes (see
    :setting:`SCHEDULER_PRIORITY_QUEUE`) now needs to support a keyword-only
    ``start_queue_cls`` parameter, not supporting it was deprecated in Scrapy
    2.13.0.
    (:gh:`7487`)

-   ``scrapy.spiders.init.InitSpider``, deprecated in Scrapy 2.13.0, is
    removed.
    (:gh:`7487`)

New features
~~~~~~~~~~~~

-   New features and improvements for
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`:

    - Support for proxies.

    - Support for the :reqmeta:`download_latency` meta key.

    - Support for :attr:`Response.certificate
      <scrapy.http.Response.certificate>`.

    - Default headers set by the ``httpx`` library are no longer added to
      requests.

    (:gh:`7441`, :gh:`7524`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    skips HTTPS proxy certificate verification when the
    :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting is set to ``False``.
    (:gh:`7496`)

Improvements
~~~~~~~~~~~~

-   :func:`time.monotonic` is used instead of :func:`time.time` to calculate
    elapsed time in various places.
    (:gh:`7377`)

-   Improved extraction of the file extension from the URL in
    :class:`~scrapy.pipelines.files.FilesPipeline`.
    (:gh:`4225`, :gh:`7414`)

-   Other code refactoring and improvements.
    (:gh:`7401`)

Bug fixes
~~~~~~~~~

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    raises an exception when a request has an ``https://`` destination and an
    ``https://`` proxy, which is not supported by this handler. Previously it
    tried to connect to the proxy via HTTP in this case.
    (:gh:`7496`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    raises an exception for requests with ``http://`` URLs instead of trying to
    connect, which is not supported by this handler.
    (:gh:`7496`)

-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no longer
    adds the ``:status`` pseudo-header to :attr:`Response.headers
    <scrapy.http.Response.headers>`.
    (:gh:`7441`)

-   Fixed :func:`scrapy.utils.response.open_in_browser` removing the ``<head>``
    tag when adding the ``<base>`` tag.
    (:gh:`7459`)

Documentation
~~~~~~~~~~~~~

-   Documented that
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    doesn't support HTTPS proxies for HTTPS destinations and that
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` doesn't
    support proxies at all.
    (:gh:`7496`)

-   Added an example of using
    :class:`logging.handlers.TimedRotatingFileHandler` to rotate Scrapy logs.
    (:gh:`3628`, :gh:`7501`)

-   Added a ``CITATION.cff`` file.
    (:gh:`7502`, :gh:`7519`)

-   Mentioned ``DOWNLOADER_CLIENT_TLS_METHOD`` in :ref:`bans`.
    (:gh:`5232`, :gh:`7518`)

-   Other documentation improvements and fixes.
    (:gh:`7417`,
    :gh:`7463`,
    :gh:`7472`,
    :gh:`7480`,
    :gh:`7489`,
    :gh:`7503`,
    :gh:`7507`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Added tests that connect to https://books.toscrape.com/ to test the
    behavior with a real website. These tests are marked with the
    ``requires_internet`` pytest mark and can be skipped with e.g.
    ``-m 'not requires_internet'`` if you cannot or don't want to run them.
    (:gh:`7520`)

-   Type hints improvements and fixes.
    (:gh:`7492`, :gh:`7532`)

-   CI and test improvements and fixes.
    (:gh:`7441`, :gh:`7466`, :gh:`7491`, :gh:`7496`)

.. _release-2.15.2:

Scrapy 2.15.2 (2026-04-28)
--------------------------

Bug fixes
~~~~~~~~~

-   Fixed links in https://docs.scrapy.org/llms.txt (:gh:`7467`)

.. _release-2.15.1:

Scrapy 2.15.1 (2026-04-23)
--------------------------

Bug fixes
~~~~~~~~~

-   Sharing of the SSL context between multiple connections, introduced in
    Scrapy 2.15.0, is reverted as it caused problems and wasn't actually
    needed.
    (:gh:`7445`, :gh:`7450`)

-   Fixed :meth:`scrapy.settings.BaseSettings.getwithbase` failing on keys with
    dots that aren't import names. It now works the way it worked before Scrapy
    2.15.0, without trying to match class objects and import path. A separate
    method,
    :func:`~scrapy.settings.BaseSettings.get_component_priority_dict_with_base`,
    was added that does that, and it is now used for :ref:`component priority
    dictionaries <component-priority-dictionaries>`.
    (:gh:`7426`, :gh:`7449`)

-   Documentation rendering improvements.
    (:gh:`7452`, :gh:`7454`)

.. _release-2.15.0:

Scrapy 2.15.0 (2026-04-09)
--------------------------

Highlights:

-   Experimental support for running without a Twisted reactor

-   Experimental ``httpx``-based download handler

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   The built-in HTTP :ref:`download handlers <download-handlers-ref>` now
    raise Scrapy-specific exceptions instead of implementation-specific ones,
    see :ref:`download-handlers-exceptions`. This can affect user code that
    handles downloader exceptions, such as ``process_exception()`` methods of
    custom :ref:`downloader middlewares <topics-downloader-middleware-custom>`.
    (:gh:`7208`)

-   In order to fix a long-standing bug with handling of asynchronous storages,
    the following changes were made to media pipeline classes, which can impact
    some of the user code that subclasses them or calls their methods directly:

    - overrides of :meth:`scrapy.pipelines.media.MediaPipeline.media_downloaded`
      and :meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded` can now
      return coroutines

    - :meth:`~scrapy.pipelines.files.FilesPipeline.media_downloaded`,
      :meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded` and
      :meth:`~scrapy.pipelines.images.ImagesPipeline.image_downloaded` now
      return coroutines

    (:gh:`2183`, :gh:`6369`, :gh:`7182`)

-   ``Request`` and ``Response`` objects: ``__slots__`` and setter changes:

    -   :class:`scrapy.http.Request` and :class:`scrapy.http.Response` now
        define ``__slots__``. Assigning arbitrary attributes to instances (for
        example, ``response.foo = 1``) will raise ``AttributeError``. Store
        per-request/response data in the request/response ``meta`` mapping
        instead of attaching new attributes to the objects.

    -   If you maintain custom ``Request`` or ``Response`` subclasses that
        relied on dynamic instance attributes, either add ``'__dict__'`` to
        your subclass ``__slots__`` to allow dynamic attributes, or migrate
        per-instance state to ``meta`` or explicit documented attributes.

    -   The setters for ``headers``, ``flags`` and ``cookies`` no longer coerce
        falsy values into ``None``. For example, ``request.headers = {}`` now
        stores an empty :class:`scrapy.http.headers.Headers` instance (not
        ``None``), and ``request.flags = []`` remains an empty list instead of
        being set to ``None``. Update code that relied on ``is None`` checks or
        the previous coercion behaviour.

    (:gh:`7036`, :gh:`7367`, :gh:`7374`)

Deprecation removals
~~~~~~~~~~~~~~~~~~~~

-   The context factory class set as the value of the
    ``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is now required to support the
    ``method`` argument of ``__init__()``, recommended since Scrapy 1.2.0.
    (:gh:`7353`)

Deprecations
~~~~~~~~~~~~

-   ``scrapy.mail.MailSender`` is deprecated. Please use :mod:`smtplib`,
    :mod:`twisted.mail.smtp` or other 3rd party email libraries.
    (:gh:`7249`, :gh:`7263`)

-   The ``scrapy.extensions.statsmailer.StatsMailer`` extension is deprecated.
    You can instead implement your own notifications by handling the
    :signal:`spider_closed` signal.
    (:gh:`7249`, :gh:`7263`)

-   The ``MEMUSAGE_NOTIFY_MAIL`` setting is deprecated. You can instead
    implement your own notifications by handling the
    :signal:`memusage_warning_reached` and :signal:`spider_closed` signals.
    (:gh:`7249`, :gh:`7263`)

-   The ``DNS_RESOLVER`` setting was renamed to :setting:`TWISTED_DNS_RESOLVER`
    and the old name is deprecated.
    (:gh:`7350`, :gh:`7361`)

-   The ``DOWNLOADER_CLIENTCONTEXTFACTORY`` setting is deprecated. If you were
    using it to switch to
    ``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory``, please
    use the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting instead. If you
    cannot use the default context factory for some other reason, please
    subclass the :ref:`download handler <download-handlers-ref>` instead.
    (:gh:`7352`, :gh:`7379`)

-   ``scrapy.core.downloader.contextfactory.BrowserLikeContextFactory`` is
    deprecated. You can set the new :setting:`DOWNLOAD_VERIFY_CERTIFICATES`
    setting to ``True`` instead.
    (:gh:`7379`)

-   The following implementation details of the context factory handling code
    are deprecated:

    - ``scrapy.core.downloader.contextfactory.AcceptableProtocolsContextFactory``

    - ``scrapy.core.downloader.contextfactory.load_context_factory_from_settings()``

    - ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory``

    - ``scrapy.core.downloader.tls.ScrapyClientTLSOptions``

    (:gh:`7353`, :gh:`7391`)

-   Passing :class:`str` instead of :class:`bytes` to
    :class:`scrapy.utils.sitemap.Sitemap` and
    :func:`scrapy.utils.sitemap.sitemap_urls_from_robots` is deprecated.
    (:gh:`7007`)

-   ``scrapy.utils.misc.walk_modules()`` is deprecated. You can use
    :func:`scrapy.utils.misc.walk_modules_iter` instead.
    (:gh:`7388`)

-   ``scrapy.shell.Shell.inthread`` is deprecated. You can use
    :attr:`scrapy.shell.Shell.fetch_available` instead to check if
    :func:`~scrapy.shell.Shell.fetch` can be used.
    (:gh:`7395`)

-   ``scrapy.commands.ScrapyCommand.set_crawler()`` is deprecated.
    (:gh:`7276`)

New features
~~~~~~~~~~~~

-   Added an *experimental* mode for running Scrapy without installing a
    Twisted reactor: set :setting:`TWISTED_REACTOR_ENABLED` to ``False`` to
    enable it. This mode has limitations, refer to :ref:`its documentation
    <asyncio-without-reactor>` for details. As long as it's experimental, its
    behavior and related features and APIs may change in future Scrapy releases
    in a breaking way.
    (:gh:`6219`,
    :gh:`7185`,
    :gh:`7186`,
    :gh:`7187`,
    :gh:`7188`,
    :gh:`7190`,
    :gh:`7197`,
    :gh:`7199`,
    :gh:`7209`,
    :gh:`7228`,
    :gh:`7355`,
    :gh:`7366`,
    :gh:`7385`,
    :gh:`7395`)

-   Added the :func:`scrapy.utils.reactorless.is_reactorless` function that
    checks if there is a running asyncio event loop but no Twisted reactor.
    (:gh:`7185`, :gh:`7199`)

-   Changed :func:`scrapy.utils.asyncio.is_asyncio_available` to return
    ``True`` if there is a running asyncio loop, even if no Twisted reactor is
    installed.
    (:gh:`7185`, :gh:`7199`)

-   Added an *experimental* download handler that uses the httpx_ library and
    doesn't require a Twisted reactor:
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. As
    long as it's experimental, its behavior may change in future Scrapy
    releases in a breaking way.
    (:gh:`6805`, :gh:`7239`, :gh:`7368`, :gh:`7384`)

    .. _httpx: https://www.python-httpx.org/

-   Added the :setting:`DOWNLOAD_BIND_ADDRESS` setting as a global counterpart
    to the per-request :reqmeta:`bindaddress` meta key.
    (:gh:`7266`, :gh:`7283`)

-   Added the :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting that can be set
    to ``True`` to make Scrapy abort HTTPS requests when the server certificate
    is invalid or doesn't match the domain.
    (:gh:`7379`)

-   The built-in HTTP :ref:`download handlers <download-handlers-ref>` now
    raise Scrapy-specific exceptions instead of implementation-specific ones,
    to allow unified handling of similar problems caused by different
    implementations. The default value of the :setting:`RETRY_EXCEPTIONS`
    setting was updated replacing Twisted-specific exceptions with these new
    ones. The exceptions:

    - :exc:`~scrapy.exceptions.CannotResolveHostError`

    - :exc:`~scrapy.exceptions.DownloadCancelledError`

    - :exc:`~scrapy.exceptions.DownloadConnectionRefusedError`

    - :exc:`~scrapy.exceptions.DownloadFailedError`

    - :exc:`~scrapy.exceptions.DownloadTimeoutError`

    - :exc:`~scrapy.exceptions.ResponseDataLossError`

    - :exc:`~scrapy.exceptions.UnsupportedURLSchemeError`

    (:gh:`7208`)

-   Added the :signal:`memusage_warning_reached` signal emitted by the
    :class:`~scrapy.extensions.memusage.MemoryUsage` extension when the memory
    usage reaches :setting:`MEMUSAGE_WARNING_MB`.
    (:gh:`7249`, :gh:`7263`)

-   Added
    :meth:`Headers.to_tuple_list() <scrapy.http.headers.Headers.to_tuple_list>`
    that returns headers as a list of ``(key, value)`` tuples.
    (:gh:`7239`)

-   :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler` now uses the
    download handler configured for the ``"https"`` scheme to make requests
    instead of always using
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
    (:gh:`7369`, :gh:`7370`)

-   Added :func:`scrapy.utils.misc.walk_modules_iter` as a replacement for
    ``scrapy.utils.misc.walk_modules()`` that returns an iterable instead of a
    list.
    (:gh:`7388`)

Improvements
~~~~~~~~~~~~

-   :func:`asyncio.to_thread` is now used instead of
    :func:`twisted.internet.threads.deferToThread` in the built-in feed
    storages, media pipeline storages and the
    :func:`scrapy.utils.decorators.inthread` decorator when available.
    (:gh:`7183`, :gh:`7184`, :gh:`7349`)

-   Improved memory footprint of :class:`~scrapy.Request` and
    :class:`~scrapy.http.Response` objects by adding ``__slots__`` and omitting
    empty lists and dicts in some internal attributes.
    (:gh:`7036`, :gh:`7367`, :gh:`7374`)

-   :class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`
    no longer mutates the SSL context, to avoid the behavior that was
    deprecated in pyOpenSSL 25.1.0.
    (:gh:`6859`, :gh:`7353`)

-   Improved memory usage of :class:`~scrapy.spiders.sitemap.SitemapSpider` and
    :class:`scrapy.utils.sitemap.Sitemap`.
    (:gh:`3529`, :gh:`7007`)

-   Improved the scheduling behavior of
    :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue` when crawling
    multiple domains.
    (:gh:`7293`, :gh:`7351`)

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now handle
    TLS verbose logging (see :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING`)
    directly instead of relying on
    :class:`~scrapy.core.downloader.contextfactory._ScrapyClientContextFactory`.
    (:gh:`7387`)

-   The server certificate verification code now correctly handles certificates
    with IP addresses in ``subjectAltName``.
    (:gh:`7353`)

-   Improved reliability of :func:`scrapy.utils.trackref.get_oldest`.
    (:gh:`1758`, :gh:`7375`)

-   Other code refactoring and improvements.
    (:gh:`7210`, :gh:`7238`, :gh:`7376`, :gh:`7386`, :gh:`7395`,
    :gh:`7405`, :gh:`7410`)

Bug fixes
~~~~~~~~~

-   :ref:`Media pipelines <topics-media-pipeline>` should now wait for uploads
    to asynchronous storages (e.g.
    :class:`~scrapy.pipelines.files.S3FilesStore`) to complete.
    (:gh:`2183`, :gh:`6369`, :gh:`7182`)

-   Fixed merging ``*_BASE`` settings (e.g. merging
    :setting:`DOWNLOADER_MIDDLEWARES` with
    :setting:`DOWNLOADER_MIDDLEWARES_BASE`) when a component is referred to by
    a class object in one setting and by a string import path in the other one.
    (:gh:`6912`, :gh:`6993`)

-   ``scrapy runspider`` and ``scrapy crawl`` now set the exit code to 1 if an
    exception happened early (this was broken since Scrapy 2.13.0).
    (:gh:`6820`, :gh:`7255`)

-   Fixed repeated warnings about data loss (see
    :setting:`DOWNLOAD_FAIL_ON_DATALOSS`) not being suppressed in
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
    (:gh:`7222`)

-   Improved FTP connection management in
    :class:`scrapy.pipelines.files.FTPFilesStore`.
    (:gh:`7256`)

-   Fixed the ``spider`` variable in the :ref:`shell <topics-shell>`, which
    wasn't available since Scrapy 2.13.0.
    (:gh:`7395`)

Documentation
~~~~~~~~~~~~~

-   The ``llms.txt`` and ``llms-full.txt`` files and Markdown versions of pages
    are now generated when the HTML documentation is built.
    (:gh:`7380`)

-   Added a "Copy as Markdown" button to the HTML documentation.
    (:gh:`7380`)

-   Added :ref:`docs for using Pydantic models as items <pydantic-items>`.
    (:gh:`6955`, :gh:`6966`)

-   Documented :ref:`job directory contents <job-dir-contents>`.
    (:gh:`4842`, :gh:`5260`)

-   Improved docs for :attr:`~scrapy.Request.dont_filter`.
    (:gh:`6398`, :gh:`7245`)

-   Clarified that settings related to :setting:`TWISTED_DNS_RESOLVER` are only
    taken into account if the selected resolver supports them.
    (:gh:`7385`)

-   Other documentation improvements and fixes.
    (:gh:`7248`, :gh:`7274`, :gh:`7406`, :gh:`7408`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Added the ``no-reactor`` test environment that doesn't install a Twisted
    reactor and uses ``pytest-asyncio`` instead of ``pytest-twisted`` to run
    asynchronous test functions.
    (:gh:`6952`, :gh:`7189`, :gh:`7233`, :gh:`7234`, :gh:`7254`,
    :gh:`7259`)

-   Fixed running tests with ``pytest-xdist``.
    (:gh:`7216`, :gh:`7257`)

-   Type hints improvements and fixes.
    (:gh:`7300`, :gh:`7331`)

-   CI and test improvements and fixes.
    (:gh:`7060`,
    :gh:`7223`,
    :gh:`7232`,
    :gh:`7241`,
    :gh:`7250`,
    :gh:`7256`,
    :gh:`7276`,
    :gh:`7277`,
    :gh:`7279`,
    :gh:`7329`,
    :gh:`7363`,
    :gh:`7381`,
    :gh:`7402`)

.. _release-2.14.2:

Scrapy 2.14.2 (2026-03-12)
--------------------------

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Values from the ``Referrer-Policy`` header of HTTP responses are no longer
    executed as Python callables. See the `cwxj-rr6w-m6w7`_ security advisory
    for details.

    .. _cwxj-rr6w-m6w7: https://github.com/scrapy/scrapy/security/advisories/GHSA-cwxj-rr6w-m6w7

-   In line with the `standard
    <https://fetch.spec.whatwg.org/#http-redirect-fetch>`__, 301 redirects of
    ``POST`` requests are converted into ``GET`` requests.

    Converting to a ``GET`` request implies not only a method change, but also
    omitting the body and ``Content-*`` headers in the redirect request. On
    cross-origin redirects (for example, cross-domain redirects), this is
    effectively a security bug fix for scenarios where the body contains
    secrets.

Deprecations
~~~~~~~~~~~~

-   Passing a response URL string as the first positional argument to
    :meth:`scrapy.spidermiddlewares.referer.RefererMiddleware.policy` is
    deprecated. Pass a :class:`~scrapy.http.Response` instead.

    The parameter has also been renamed to ``response`` to reflect this change.
    The old parameter name (``resp_or_url``) is deprecated.

New features
~~~~~~~~~~~~

-   Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing
    supported referrer policies.

Bug fixes
~~~~~~~~~

-   Made additional redirect scenarios convert to ``GET`` in line with the
    `standard <https://fetch.spec.whatwg.org/#http-redirect-fetch>`__:

    -   Only ``POST`` 302 redirects are converted into ``GET`` requests; other
        methods are preserved.

    -   ``HEAD`` 303 redirects are not converted into ``GET`` requests.

    -   ``GET`` 303 redirects do not have their body or standard ``Content-*``
        headers removed.

-   Redirects where the original request body is dropped now also have their
    ``Content-Encoding``, ``Content-Language`` and ``Content-Location`` headers
    removed, in addition to the ``Content-Type`` and ``Content-Length`` headers
    that were already being removed.

-   Redirects now preserve the source URL fragment if the redirect URL does not
    include one. This is useful when using browser-based download handlers,
    such as `scrapy-playwright`_ or `scrapy-zyte-api`_, while letting Scrapy
    handle redirects.

    .. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
    .. _scrapy-zyte-api: https://scrapy-zyte-api.readthedocs.io/en/latest/

-   The ``Referer`` header is now removed on redirect if
    :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` is disabled.

-   The handling of the ``Referer`` header on redirects now takes into account
    the ``Referer-Policy`` header of the response that triggers the redirect.

.. _release-2.14.1:

Scrapy 2.14.1 (2026-01-12)
--------------------------

Deprecations
~~~~~~~~~~~~

-   ``scrapy.utils.defer.maybeDeferred_coro()`` is deprecated. (:gh:`7212`)

Bug fixes
~~~~~~~~~

-   Fixed custom stats collectors that require a ``spider`` argument in their
    ``open_spider()`` and ``close_spider()`` methods not receiving the
    argument when called by the engine.

    Note, however, that the ``spider`` argument is now deprecated and will stop
    being passed in a future version of Scrapy.

    (:gh:`7213`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Replaced deprecated ``codecov/test-results-action@v1`` GitHub Action with
    ``codecov/codecov-action@v5``.
    (:gh:`7180`, :gh:`7215`)

.. _release-2.14.0:

Scrapy 2.14.0 (2026-01-05)
--------------------------

Highlights:

-   More coroutine-based replacements for Deferred-based APIs

-   The default priority queue is now ``DownloaderAwarePriorityQueue``

-   Dropped support for Python 3.9 and PyPy 3.10

-   Improved and documented the API for custom download handlers

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   Dropped support for Python 3.9.
    (:gh:`7121`)

-   Dropped support for PyPy 3.10.
    (:gh:`7050`)

-   Increased the minimum versions of the following dependencies:

    - lxml_: 4.6.0 → 4.6.4

    - Pillow_ (optional dependency): 8.0.0 → 8.3.2

    - botocore_ (optional dependency): 1.4.87 → 1.13.45

-   Restored support for ``brotlicffi`` dropped in Scrapy 2.13.4. Its minimum
    supported version is now ``1.2.0.0``.
    (:gh:`7160`)

Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-   If you set the ``TWISTED_REACTOR`` setting to a non-asyncio
    value at the :ref:`spider level <spider-settings>`, you
    may now need to set the ``FORCE_CRAWLER_PROCESS`` setting to
    ``True`` when running Scrapy via :ref:`its command-line tool
    <topics-commands-crawlerprocess>` to avoid a reactor mismatch exception.
    (:gh:`6845`)

-   The ``log_count/*`` stats no longer count some of the early messages that
    they counted before. While the earliest log messages, emitted before the
    counter is initialized, were never counted, the counter initialization now
    happens later than in previous Scrapy versions. You may need to adjust
    expected values if you retrieve and compare values of these stats in your
    code.
    (:gh:`7046`)

-   The classes listed below are now :term:`abstract base classes <abstract
    base class>`. They cannot be instantiated directly and their subclasses
    need to override the abstract methods listed below to be able to be
    instantiated. If you previously instantiated these classes directly, you
    will now need to subclass them and provide trivial (e.g. empty)
    implementations for the abstract methods.

    - :class:`scrapy.commands.ScrapyCommand`

        - :meth:`~scrapy.commands.ScrapyCommand.run`

        - :meth:`~scrapy.commands.ScrapyCommand.short_desc`

    - :class:`scrapy.exporters.BaseItemExporter`

        - :meth:`~scrapy.exporters.BaseItemExporter.export_item`

    - :class:`scrapy.extensions.feedexport.BlockingFeedStorage`

        - :meth:`~scrapy.extensions.feedexport.BlockingFeedStorage._store_in_thread`

    - :class:`scrapy.middleware.MiddlewareManager`

        - :meth:`~scrapy.middleware.MiddlewareManager._get_mwlist_from_settings`

    - :class:`scrapy.spidermiddlewares.referer.ReferrerPolicy`

        - :meth:`~scrapy.spidermiddlewares.referer.ReferrerPolicy.referrer`

    (:gh:`6930`)

-   Scrapy no longer passes a ``spider`` argument to any methods of the
    :setting:`stats collector <STATS_CLASS>`. It wasn't passed in many of the
    calls even in older Scrapy versions, so we don't expect existing custom
    stats collector implementations to require a ``spider`` argument. If your
    implementation needs a :class:`~scrapy.Spider` instance, you can get it
    from the :class:`~scrapy.crawler.Crawler` instance passed to the
    constructor.
    (:gh:`7011`)

-   :class:`scrapy.middleware.MiddlewareManager` no longer includes code for
    handling ``open_spider()`` and ``close_spider()`` component methods. As
    this code was only used for pipelines it was moved into
    :class:`scrapy.pipelines.ItemPipelineManager`. This change should only
    affect custom subclasses of :class:`~scrapy.middleware.MiddlewareManager`.
    The following code was moved:

    - ``scrapy.middleware.MiddlewareManager.open_spider()``

    - ``scrapy.middleware.MiddlewareManager.close_spider()``

    - Code in ``scrapy.middleware.MiddlewareManager._add_middleware()`` that
      processes ``open_spider()`` and ``close_spider()`` component methods.

    (:gh:`7006`)

-   :meth:`scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware.process_request`
    now returns a coroutine, previously it returned a
    :class:`~twisted.internet.defer.Deferred` object or ``None``. The
    ``robot_parser()`` method was also changed to return a coroutine. This
    change only impacts code that subclasses
    :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` or
    calls its methods directly.
    (:gh:`6802`)

-   The built-in :ref:`download handlers <download-handlers-ref>` have been
    refactored, changing the signatures of their methods. This change should
    only affect user code that subclasses any of these handlers or calls their
    methods directly.
    (:gh:`6778`, :gh:`7164`)

-   :meth:`scrapy.pipelines.media.MediaPipeline.process_item` now returns a
    coroutine, previously it returned a
    :class:`~twisted.internet.defer.Deferred` object. This
    change only impacts code that calls this method directly.
    (:gh:`7177`)

Deprecation removals
~~~~~~~~~~~~~~~~~~~~

-   The ``from_settings()`` method of the following components, deprecated in
    Scrapy 2.12.0, is removed. You should use ``from_crawler()`` instead.

    - :class:`scrapy.dupefilters.RFPDupeFilter`
    - :class:`scrapy.mail.MailSender`
    - :class:`scrapy.middleware.MiddlewareManager`
    - :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`
    - :class:`scrapy.pipelines.files.FilesPipeline`
    - :class:`scrapy.pipelines.images.ImagesPipeline`

    (:gh:`7126`)

-   Scrapy no longer calls ``from_settings()`` methods of 3rd-party
    :ref:`components <topics-components>`, deprecated in Scrapy 2.12.0. You
    should define a ``from_crawler()`` method instead.
    (:gh:`7126`)

-   The initialization flow of :class:`scrapy.pipelines.media.MediaPipeline`
    and its subclasses was simplified, it now mandates ``from_crawler()``
    methods and ``crawler`` arguments of ``__init__()`` methods. Not using
    these was deprecated in Scrapy 2.12.0.
    (:gh:`7126`)

-   The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting, deprecated in Scrapy
    2.12.0, is removed.
    (:gh:`7126`)

-   The ``scrapy.utils.misc.create_instance()`` function, deprecated in Scrapy
    2.12.0, is removed. Use :func:`scrapy.utils.misc.build_from_crawler`
    instead.
    (:gh:`7126`)

-   The ``scrapy.core.downloader.Downloader._get_slot_key()`` function,
    deprecated in Scrapy 2.12.0, is removed. Use
    :meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
    (:gh:`7126`)

-   The ``scrapy.twisted_version`` attribute, deprecated in Scrapy 2.12.0, is
    removed. You should instead use the :attr:`twisted.version` attribute
    directly.
    (:gh:`7126`)

-   The following utility functions, deprecated in Scrapy 2.12.0, are removed:

    - ``scrapy.utils.defer.process_chain_both()``
    - ``scrapy.utils.python.equal_attributes()``
    - ``scrapy.utils.python.flatten()``
    - ``scrapy.utils.python.iflatten()``
    - ``scrapy.utils.request.request_authenticate()``
    - ``scrapy.utils.test.assert_samelines()``

    (:gh:`7126`)

-   ``scrapy.utils.serialize.ScrapyJSONDecoder``, deprecated in Scrapy 2.12.0,
    is removed.
    (:gh:`7126`)

-   The ``scrapy.extensions.feedexport.build_storage()`` function, deprecated
    in Scrapy 2.12.0, is removed, you can instead call the builder callable
    directly.
    (:gh:`7126`)

-   ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware``, deprecated in
    Scrapy 2.11.2, is removed.
    :class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` should be
    used instead.
    (:gh:`6926`)

Deprecations
~~~~~~~~~~~~

-   The following methods that return a
    :class:`~twisted.internet.defer.Deferred` are deprecated in favor of their
    coroutine-based replacements:

    - :class:`scrapy.core.downloader.handlers.DownloadHandlers`

        - ``download_request()`` (use
          :meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`)

    - :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`

        - ``download()`` (use
          :meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`)

    - :class:`scrapy.core.engine.ExecutionEngine`

        - ``start()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.start_async`)

        - ``stop()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.stop_async`)

        - ``close()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.close_async`)

        - ``open_spider()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`)

        - ``close_spider()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`)

        - ``download()`` (use
          :meth:`~scrapy.core.engine.ExecutionEngine.download_async`)

    - :class:`scrapy.core.scraper.Scraper`

        - ``open_spider()`` (use
          :meth:`~scrapy.core.scraper.Scraper.open_spider_async`)

        - ``call_spider()`` (use
          :meth:`~scrapy.core.scraper.Scraper.call_spider_async`)

        - ``close_spider()`` (use
          :meth:`~scrapy.core.scraper.Scraper.close_spider_async`)

        - ``handle_spider_output()`` (use
          :meth:`~scrapy.core.scraper.Scraper.handle_spider_output_async`)

        - ``start_itemproc()`` (use
          :meth:`~scrapy.core.scraper.Scraper.start_itemproc_async`)

    - :class:`scrapy.core.spidermw.SpiderMiddlewareManager`

        - ``scrape_response()`` (use
          :meth:`~scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response_async`)

    - :class:`scrapy.crawler.Crawler`

        - ``stop()`` (use :meth:`~scrapy.crawler.Crawler.stop_async`)

    - :class:`scrapy.pipelines.ItemPipelineManager`

        - ``process_item()`` (use
          :meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async`)

        - ``open_spider()`` (use
          :meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async`)

        - ``close_spider()`` (use
          :meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async`)

    - :class:`scrapy.signalmanager.SignalManager`

        - ``send_catch_log_deferred()`` (use
          :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async`)

    - ``scrapy.utils.signal.send_catch_log_deferred()`` (use
      :func:`scrapy.utils.signal.send_catch_log_async`)

    (:gh:`6791`, :gh:`6842`, :gh:`6979`, :gh:`6997`, :gh:`6999`,
    :gh:`7005`, :gh:`7043`, :gh:`7069`, :gh:`7161`, :gh:`7164`)

-   The following spider attributes are deprecated in favor of settings:

    - ``download_maxsize`` (use :setting:`DOWNLOAD_MAXSIZE`)

    - ``download_timeout`` (use :setting:`DOWNLOAD_TIMEOUT`)

    - ``download_warnsize`` (use :setting:`DOWNLOAD_WARNSIZE`)

    - ``max_concurrent_requests`` (use
      :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`)

    - ``user_agent`` (use :setting:`USER_AGENT`)

    (:gh:`6988`, :gh:`6994`, :gh:`7038`, :gh:`7039`, :gh:`7117`,
    :gh:`7176`)

-   Returning a :class:`~twisted.internet.defer.Deferred` from the following
    user-defined functions is deprecated in favor of defining them as coroutine
    functions:

    - spider callbacks and errbacks (which was never officially supported and
      may work incorrectly)

    - the ``process_request()``, ``process_response()`` and
      ``process_exception()`` methods of custom downloader middlewares

    - the ``process_item()``, ``open_spider()`` and ``close_spider()`` methods
      of custom pipelines

    - signal handlers

    - the ``download_request()`` and ``close()`` methods of custom download
      handlers

    (:gh:`6718`, :gh:`6778`, :gh:`7069`, :gh:`7147`, :gh:`7148`,
    :gh:`7149`, :gh:`7150`, :gh:`7151`, :gh:`7161`, :gh:`7164`,
    :gh:`7179`)

-   Passing a ``spider`` argument to the following methods is deprecated:

    - :meth:`scrapy.core.spidermw.SpiderMiddlewareManager.process_start`

    - :meth:`scrapy.core.downloader.Downloader.fetch`

    - :meth:`scrapy.core.downloader.Downloader._get_slot`

    - :meth:`scrapy.core.downloader.handlers.DownloadHandlers.download_request`

    - all public methods of :class:`scrapy.statscollectors.StatsCollector`

    - :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output`

    - :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output_async`

    - all ``process_*()`` methods of built-in downloader middlewares

    - all ``process_*()`` methods of built-in spider middlewares

    - :meth:`scrapy.pipelines.media.MediaPipeline.open_spider`

    - :meth:`scrapy.pipelines.media.MediaPipeline.process_item`

    (:gh:`6750`, :gh:`6927`, :gh:`6984`, :gh:`7006`, :gh:`7011`,
    :gh:`7033`, :gh:`7037`, :gh:`7045`, :gh:`7178`)

-   Instantiating subclasses of :class:`scrapy.middleware.MiddlewareManager`
    without a :class:`~scrapy.crawler.Crawler` instance is deprecated.
    (:gh:`6984`)

-   For the following user-defined functions and methods requiring a ``spider``
    argument is deprecated, if you need a :class:`~scrapy.Spider` instance
    inside them you should get it from the :class:`~scrapy.crawler.Crawler`
    instance (you may need to refactor your code to save that instance in e.g.
    the ``from_crawler()`` method):

    - the ``process_request()``, ``process_response()`` and
      ``process_exception()`` methods of custom downloader middlewares

    - the ``process_spider_input()``, ``process_spider_output()``,
      ``process_spider_output_async()`` and ``process_spider_exception()``
      methods of custom spider middlewares

    - the ``process_item()`` method of custom pipelines

    - the ``fetch()`` method of a custom :setting:`DOWNLOADER`

    (:gh:`6927`, :gh:`6984`, :gh:`7006`, :gh:`7037`)

-   The following things in custom download handlers are deprecated:

    - not having a ``lazy`` attribute (you should define it as ``True`` if you
      want to keep the current behavior)

    - returning a :class:`~twisted.internet.defer.Deferred` from the
      ``download_request()`` method (you should refactor it to return a
      coroutine; you also need to remove the ``spider`` argument when doing
      this)

    - not having a ``close()`` method, having a synchronous one or one that
      returns a :class:`~twisted.internet.defer.Deferred` (you should refactor
      it to return a coroutine or add an empty one if you don't have it)

    (:gh:`6778`, :gh:`7164`)

-   Custom implementations of :setting:`ITEM_PROCESSOR` should now define
    ``process_item_async()``, ``open_spider_async()`` and
    ``close_spider_async()`` methods instead of, or in addition to,
    ``process_item()``, ``open_spider()`` and ``close_spider()``.
    (:gh:`7005`, :gh:`7043`)

-   The ``CONCURRENT_REQUESTS_PER_IP`` setting is deprecated, use
    :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` instead.
    (:gh:`6917`, :gh:`6921`)

-   The ``scrapy.core.downloader.handlers.http`` module is deprecated. You
    should import
    :class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    directly instead of importing the
    ``scrapy.core.downloader.handlers.http.HTTPDownloadHandler`` alias.
    (:gh:`7079`)

-   The ``scrapy.utils.decorators.defers()`` decorator is deprecated, you can
    use :func:`twisted.internet.defer.maybeDeferred` directly or reimplement
    this decorator in your code.
    (:gh:`7164`)

-   ``scrapy.spiders.CrawlSpider._parse_response()`` is deprecated, use
    :meth:`scrapy.spiders.CrawlSpider.parse_with_rules` instead.
    (:gh:`4463`, :gh:`6804`)

-   The functions that add a delay to a Deferred are deprecated, their
    underlying Twisted functions can be used instead, either directly if a
    delay isn't needed, or with some explicit way to add a delay if it's
    needed:

    - ``scrapy.utils.defer.mustbe_deferred()`` (you can use
      :func:`twisted.internet.defer.maybeDeferred`)

    - ``scrapy.utils.defer.defer_succeed()`` (you can use
      :func:`twisted.internet.defer.succeed`)

    - ``scrapy.utils.defer.defer_fail()`` (you can use
      :func:`twisted.internet.defer.fail`)

    - ``scrapy.utils.defer.defer_result()`` (you can use
      :func:`twisted.internet.defer.succeed` and
      :func:`twisted.internet.defer.fail`)

    (:gh:`6937`)

New features
~~~~~~~~~~~~

-   Added :class:`scrapy.crawler.AsyncCrawlerProcess` and
    :class:`scrapy.crawler.AsyncCrawlerRunner` as counterparts to
    :class:`~scrapy.crawler.CrawlerProcess` and
    :class:`~scrapy.crawler.CrawlerRunner` that offer coroutine-based APIs.
    (:gh:`6789`, :gh:`6790`, :gh:`6796`, :gh:`6817`, :gh:`6845`,
    :gh:`7034`)

-   Added coroutine counterparts to some of the Deferred-based APIs:

    - :class:`scrapy.core.downloader.handlers.DownloadHandlers`

        - :meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`
          (to ``download_request()``)

    - :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`

        - :meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`
          (to ``download()``)

    - :class:`scrapy.core.engine.ExecutionEngine`

        - :meth:`~scrapy.core.engine.ExecutionEngine.start_async` (to
          ``start()``)

        - :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` (to
          ``stop()``)

        - :meth:`~scrapy.core.engine.ExecutionEngine.close_async` (to
          ``close()``)

        - :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` (to
          ``open_spider()``)

        - :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` (to
          ``close_spider()``)

        - :meth:`~scrapy.core.engine.ExecutionEngine.download_async` (to
          ``download()``)

    - :class:`scrapy.core.scraper.Scraper`

        - :meth:`~scrapy.core.scraper.Scraper.open_spider_async` (to
          ``open_spider()``)

        - :meth:`~scrapy.core.scraper.Scraper.close_spider_async` (to
          ``close_spider()``)

        - :meth:`~scrapy.core.scraper.Scraper.start_itemproc_async` (to
          ``start_itemproc()``)

    - :class:`scrapy.crawler.Crawler`

        - :meth:`~scrapy.crawler.Crawler.crawl_async` (to ``crawl()``)

        - :meth:`~scrapy.crawler.Crawler.stop_async` (to ``stop()``)

    - :class:`scrapy.pipelines.ItemPipelineManager`

        - :meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async` (to
          ``process_item()``)

        - :meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async` (to
          ``open_spider()``)

        - :meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async` (to
          ``close_spider()``)

    - :class:`scrapy.signalmanager.SignalManager`

        - :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async` (to
          ``send_catch_log_deferred()``)

    (:gh:`6781`, :gh:`6791`, :gh:`6792`, :gh:`6795`, :gh:`6801`,
    :gh:`6817`, :gh:`6842`, :gh:`6997`, :gh:`7005`, :gh:`7043`,
    :gh:`7069`,:gh:`7164`, :gh:`7202`)

-   The default value of the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is now
    ``'scrapy.pqueues.DownloaderAwarePriorityQueue'``.
    (:gh:`6924`, :gh:`6940`)

-   Added :class:`scrapy.extensions.logcount.LogCount`, an enabled-by-default
    extension that is responsible for the ``log_count/*`` stats. Previously,
    this code was in :class:`scrapy.crawler.Crawler` and couldn't be disabled.
    (:gh:`7046`)

-   Added :meth:`scrapy.spiders.CrawlSpider.parse_with_rules` as a public
    replacement for ``_parse_response()``.
    (:gh:`4463`, :gh:`6804`)

-   Added :func:`scrapy.utils.asyncio.is_asyncio_available` as an alternative
    to ``scrapy.utils.reactor.is_asyncio_reactor_installed`` with a
    future-proof name and semantics.
    (:gh:`6827`)

-   The API for :ref:`download handlers <topics-download-handlers>`, previously
    undocumented, has been modernized and documented. An optional base class,
    :class:`scrapy.core.downloader.handlers.base.BaseDownloadHandler`, has been
    added to simplify writing custom download handlers that conform to the
    current API.
    (:gh:`4944`, :gh:`6778`, :gh:`7164`)

-   Added :func:`scrapy.utils.defer.ensure_awaitable`, which can be helpful to
    call user-defined functions that can return coroutines, Deferreds or
    values directly.
    (:gh:`7005`)

-   The ``requests.seen`` file, written by
    :class:`~scrapy.dupefilters.RFPDupeFilter` when :ref:`job persistence
    <topics-jobs>` is enabled, now uses line buffering to reduce data loss in
    spider crashes.
    (:gh:`6019`, :gh:`7094`)

-   Images downloaded by :class:`~scrapy.pipelines.images.ImagesPipeline` are
    now automatically transposed based on EXIF data.
    (:gh:`6525`, :gh:`6975`)

Improvements
~~~~~~~~~~~~

-   Refactored internal functions to use coroutines instead of Deferreds.
    (:gh:`6795`, :gh:`6852`, :gh:`6855`, :gh:`6858`, :gh:`7159`)

-   Commands that don't need a :class:`~scrapy.crawler.CrawlerProcess` instance
    no longer create it.
    (:gh:`6824`)

-   Improved :command:`shell` help formatting when using IPython 9+.
    (:gh:`6915`, :gh:`6980`)

Bug fixes
~~~~~~~~~

-   Setting :setting:`FILES_STORE` or :setting:`IMAGES_STORE` to ``None`` now
    correctly disables the respective pipeline.
    (:gh:`6964`, :gh:`6969`)

-   :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` now
    uses the URL set in the ``<base>`` tag as the base URL when redirecting to
    a relative URL.
    (:gh:`7042`, :gh:`7047`)

-   Passing ``None`` as a value of the :reqmeta:`download_slot` request meta
    key is now handled in the same way as not setting this meta key at all.
    (:gh:`7172`)

-   Fixed parsing of the first line of ``robots.txt`` files that have a BOM.
    (:gh:`6195`, :gh:`7095`)

Documentation
~~~~~~~~~~~~~

-   Added :ref:`documentation <topics-download-handlers>` about download
    handlers, their API and built-in handlers.
    (:gh:`4944`, :gh:`7164`)

-   Added a section about the `scrapy-spider-metadata`_ library to the
    :ref:`spider argument docs <spiderargs-scrapy-spider-metadata>`.
    (:gh:`6676`, :gh:`6957`, :gh:`7116`)

    .. _scrapy-spider-metadata: https://scrapy-spider-metadata.readthedocs.io/en/latest/

-   Improved :ref:`the docs <coroutine-deferred-apis>` about coroutine-based
    and Deferred-based APIs.
    (:gh:`6800`, :gh:`7146`)

-   Other documentation improvements and fixes.
    (:gh:`7058`, :gh:`7076`, :gh:`7109`, :gh:`7195`, :gh:`7198`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Switched from ``twisted.trial`` to ``pytest-twisted`` and replaced
    remaining ``unittest`` and ``twisted.trial`` features with ``pytest`` ones.
    (:gh:`6658`, :gh:`6873`, :gh:`6884`, :gh:`6938`)

-   Enabled fancy ``pytest`` asserts.
    (:gh:`6888`)

-   Added `Sphinx Lint`_ to the ``pre-commit`` configuration.
    (:gh:`6920`)

    .. _Sphinx Lint: https://github.com/sphinx-contrib/sphinx-lint

-   CI and test improvements and fixes.
    (:gh:`6649`,
    :gh:`6769`,
    :gh:`6821`,
    :gh:`6835`,
    :gh:`6836`,
    :gh:`6846`,
    :gh:`6883`,
    :gh:`6885`,
    :gh:`6889`,
    :gh:`6905`,
    :gh:`6928`,
    :gh:`6933`,
    :gh:`6941`,
    :gh:`6942`,
    :gh:`6945`,
    :gh:`6947`,
    :gh:`6960`,
    :gh:`6968`,
    :gh:`6972`,
    :gh:`6974`,
    :gh:`6996`,
    :gh:`7003`,
    :gh:`7012`,
    :gh:`7013`,
    :gh:`7050`,
    :gh:`7059`,
    :gh:`7070`,
    :gh:`7073`,
    :gh:`7118`,
    :gh:`7127`,
    :gh:`7141`,
    :gh:`7143`,
    :gh:`7145`,
    :gh:`7173`)

-   Code cleanups.
    (:gh:`6803`,
    :gh:`6838`,
    :gh:`6849`,
    :gh:`6875`,
    :gh:`6876`,
    :gh:`6892`,
    :gh:`6930`,
    :gh:`6949`,
    :gh:`6970`,
    :gh:`6977`,
    :gh:`6986`,
    :gh:`7008`,
    :gh:`7177`)

Older releases
--------------

.. Once the first release of a minor version series is over a year old,
   move the whole series to news/2.x.rst.

.. toctree::
   :maxdepth: 1

   news/2.x
   news/1.x
   news/0.x


.. _aioboto3: https://github.com/terricain/aioboto3
.. _aiodns: https://github.com/aio-libs/aiodns
.. _aioftp: https://github.com/aio-libs/aioftp
.. _aiohttp: https://docs.aiohttp.org/en/stable/
.. _boto3: https://github.com/boto/boto3
.. _botocore: https://github.com/boto/botocore
.. _charset-normalizer: https://charset-normalizer.readthedocs.io/en/latest/
.. _cryptography: https://cryptography.io/en/latest/
.. _cssselect: https://github.com/scrapy/cssselect
.. _formerly: https://github.com/scrapy/formerly
.. _frostwork: https://github.com/scrapy/frostwork
.. _google-cloud-storage: https://github.com/googleapis/python-storage
.. _httpx2: https://github.com/pydantic/httpx2
.. _ipdb: https://github.com/gotcha/ipdb
.. _itemloaders: https://github.com/scrapy/itemloaders
.. _lxml: https://lxml.de/
.. _parsel: https://github.com/scrapy/parsel
.. _parso: https://github.com/davidhalter/parso
.. _platformdirs: https://platformdirs.readthedocs.io/en/latest/
.. _Pillow: https://github.com/python-pillow/Pillow
.. _Protego: https://github.com/scrapy/protego
.. _ptpython: https://github.com/prompt-toolkit/ptpython
.. _pyOpenSSL: https://www.pyopenssl.org/en/stable/
.. _queuelib: https://github.com/scrapy/queuelib
.. _service_identity: https://service-identity.readthedocs.io/en/stable/
.. _Twisted: https://twisted.org/
.. _typing-extensions: https://github.com/python/typing_extensions
.. _uvloop: https://github.com/MagicStack/uvloop
.. _w3lib: https://github.com/scrapy/w3lib
.. _zope.interface: https://zopeinterface.readthedocs.io/en/latest/