Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge branch '2.11-redos' into 2.11

Adrián Chaves479619b340f1
diff --cc docs/news.rst
index ad931d6db,f346d1239..16e7e79a7
--- a/docs/news.rst
+++ b/docs/news.rst
@@@ -8,80 -8,27 +8,99 @@@ Release note
Scrapy 2.11.1 (unreleased)
--------------------------
-**Security bug fix:**
+Highlights:
+
+- Security bug fixes.
+
+- Support for Twisted >= 23.8.0.
+
+- Documentation improvements.
+
+Security bug fixes
+~~~~~~~~~~~~~~~~~~
+ - Addressed `ReDoS vulnerabilities`_:
+
+ - ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of
+ :func:`~scrapy.utils.iterators.xmliter_lxml`, which
+ :class:`~scrapy.spiders.XMLFeedSpider` now uses.
+
+ To minimize the impact of this change on existing code,
+ :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
+ the node namespace with a prefix in the node name, and big files with
+ highly nested trees when using libxml2 2.7+.
+
+ - Fixed regular expressions in the implementation of the
+ :func:`~scrapy.utils.response.open_in_browser` function.
+
+ Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
+
+ .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
+ .. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9
+
+- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
+ to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
+ advisory`_ for more information.
+
+ .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7
+
+- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
+ deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
+ removed.
+
+- The ``Authorization`` header is now dropped on redirects to a different
+ domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
+ information.
+
+ .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- The Twisted dependency is no longer restricted to < 23.8.0. (:issue:`6024`,
+ :issue:`6064`, :issue:`6142`)
+
+Bug fixes
+~~~~~~~~~
+
+- The OS signal handling code was refactored to no longer use private Twisted
+ functions. (:issue:`6024`, :issue:`6064`, :issue:`6112`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
+ changes made in the 2.11.0 release. (:issue:`6057`, :issue:`6147`)
+
+- Extended documentation for :attr:`Request.meta <scrapy.http.Request.meta>`.
+ (:issue:`5565`)
+
+- Fixed the :reqmeta:`dont_merge_cookies` documentation. (:issue:`5936`,
+ :issue:`6077`)
+
+- Added a link to Zyte's export guides to the :ref:`feed exports
+ <topics-feed-exports>` documentation. (:issue:`6183`)
+
+- Added a missing note about backward-incompatible changes in
+ :class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes.
+ (:issue:`6060`, :issue:`6081`)
+
+- Added a missing note about removing the deprecated
+ ``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes.
+ (:issue:`6056`, :issue:`6061`)
+
+- Other documentation improvements. (:issue:`6128`, :issue:`6144`,
+ :issue:`6163`, :issue:`6190`, :issue:`6192`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Added Python 3.12 to the CI configuration, re-enabled tests that were
+ disabled when the pre-release support was added. (:issue:`5985`,
+ :issue:`6083`, :issue:`6098`)
+
+- Fixed a test issue on PyPy 7.3.14. (:issue:`6204`, :issue:`6205`)
+
.. _release-2.11.0:
@@@ -2957,22 -2903,19 +2975,33 @@@ affect subclasses
Scrapy 1.8.4 (unreleased)
-------------------------
-**Security bug fix:**
+**Security bug fixes:**
+ - Due to its `ReDoS vulnerabilities`_, ``scrapy.utils.iterators.xmliter`` is
+ now deprecated in favor of :func:`~scrapy.utils.iterators.xmliter_lxml`,
+ which :class:`~scrapy.spiders.XMLFeedSpider` now uses.
+
+ To minimize the impact of this change on existing code,
+ :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
+ the node namespace as a prefix in the node name, and big files with highly
+ nested trees when using libxml2 2.7+.
+
+ Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
+
+- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
+ to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
+ advisory`_ for more information.
+
+- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, use of the
+ ``scrapy.downloadermiddlewares.decompression`` module is discouraged and
+ will trigger a warning.
+
+- The ``Authorization`` header is now dropped on redirects to a different
+ domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
+ information.
+
+ .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
+
.. _release-1.8.3: