Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge tag '2.13.4'
Andrey Rakhmatullin4cb0144b3971
diff --cc docs/news.rstindex 2a769bc1c,4bf2a504f..96e5cb2b1--- a/docs/news.rst+++ b/docs/news.rst@@@ -3,77 -3,55 +3,126 @@@Release notes=============+Scrapy VERSION (unreleased)+---------------------------++Backward-incompatible changes+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++- If you set the :setting:`TWISTED_REACTOR` setting to a :ref:`non-asyncio+ value <disable-asyncio>` at the :ref:`spider level <spider-settings>`, you+ may now need to set the :setting:`FORCE_CRAWLER_PROCESS` setting to+ ``True`` when running Scrapy via :ref:`its command-line tool+ <topics-commands-crawlerprocess>` to avoid a reactor mismatch exception.++- The ``log_count/*`` stats no longer count some of the early messages that+ they counted before. While the earliest log messages, emitted before the+ counter is initialized, were never counted, the counter initialization now+ happens later than in previous Scrapy versions. You may need to adjust+ expected values if you retrieve and compare values of these stats in your+ code.+ (:issue:`7046`)++- The classes listed below are now :term:`abstract base classes <abstract+ base class>`. They cannot be instantiated directly and their subclasses+ need to override the abstract methods listed below to be able to be+ instantiated. If you previously instantiated these classes directly, you+ will now need to subclass them and provide trivial (e.g. empty)+ implementations for the abstract methods.++ - :class:`scrapy.commands.ScrapyCommand`++ - :meth:`~scrapy.commands.ScrapyCommand.run`++ - :meth:`~scrapy.commands.ScrapyCommand.short_desc`++ - :class:`scrapy.exporters.BaseItemExporter`++ - :meth:`~scrapy.exporters.BaseItemExporter.export_item`++ - :class:`scrapy.extensions.feedexport.BlockingFeedStorage`++ - :meth:`~scrapy.extensions.feedexport.BlockingFeedStorage._store_in_thread`++ - :class:`scrapy.middleware.MiddlewareManager`++ - :meth:`~scrapy.middleware.MiddlewareManager._get_mwlist_from_settings`++ - :class:`scrapy.spidermiddlewares.referer.ReferrerPolicy`++ - :meth:`~scrapy.spidermiddlewares.referer.ReferrerPolicy.referrer`++- :class:`scrapy.middleware.MiddlewareManager` no longer includes code for+ handling ``open_spider()`` and ``close_spider()`` component methods. As+ this code was only used for pipelines it was moved into+ :class:`scrapy.pipelines.ItemPipelineManager`. This change should only+ affect custom subclasses of :class:`~scrapy.middleware.MiddlewareManager`.+ The following code was moved:++ - ``scrapy.middleware.MiddlewareManager.open_spider()``++ - ``scrapy.middleware.MiddlewareManager.close_spider()``++ - Code in ``scrapy.middleware.MiddlewareManager._add_middleware()`` that+ processes ``open_spider()`` and ``close_spider()`` component methods.++- :meth:`scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware.process_request`+ now returns a coroutine, previously it returned a+ :class:`~twisted.internet.defer.Deferred` object or ``None``. The+ ``robot_parser()`` method was also changed to return a coroutine. This+ change only impacts code that subclasses+ :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` or+ calls its methods directly.++ .. _release-2.13.4:++ Scrapy 2.13.4 (2025-11-17)+ --------------------------++ Security bug fixes+ ~~~~~~~~~~~~~~~~~~++ - Improved protection against decompression bombs in+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`+ for responses compressed using the ``br`` and ``deflate`` methods: if a+ single compressed chunk would be larger than the response size limit (see+ :setting:`DOWNLOAD_MAXSIZE`) when decompressed, decompression is no longer+ carried out. This is especially important for the ``br`` (Brotli) method+ that can provide a very high compression ratio. Please, see the+ `CVE-2025-6176`_ and `GHSA-2qfp-q593-8484`_ security advisories for more+ information.+ (:issue:`7134`)++ .. _CVE-2025-6176: https://nvd.nist.gov/vuln/detail/CVE-2025-6176+ .. _GHSA-2qfp-q593-8484: https://github.com/advisories/GHSA-2qfp-q593-8484++ Modified requirements+ ~~~~~~~~~~~~~~~~~~~~~++ - The minimum supported version of the optional ``brotli`` package is now+ ``1.2.0``.+ (:issue:`7134`)++ - The ``brotlicffi`` and ``brotlipy`` packages can no longer be used to+ decompress Brotli-compressed responses. Please install the ``brotli``+ package instead.+ (:issue:`7134`)++ Other changes+ ~~~~~~~~~~~~~++ - Restricted the maximum supported Twisted version to ``25.5.0``, as Scrapy+ currently uses some private APIs changed in later Twisted versions.+ (:issue:`7142`)++ - Stopped setting the ``COVERAGE_CORE`` environment variable in tests, it+ didn't have an effect but caused the ``coverage`` module to produce a+ warning or an error.+ (:issue:`7137`)++ - Removed the documentation build dependency on the deprecated+ ``sphinx-hoverxref`` module.+ (:issue:`6786`, :issue:`6922`).. _release-2.13.3: