Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge branch '2.14'

Adrian Chaves4e3df249f257
diff --cc docs/news.rst
index ff8e6fb08,45c2f48be..cbdef67fc
--- a/docs/news.rst
+++ b/docs/news.rst
@@@ -3,17 -3,79 +3,90 @@@
Release notes
=============
+Scrapy VERSION (unreleased)
+---------------------------
+
+New features
+~~~~~~~~~~~~
+
+- Added
+ :meth:`Headers.to_tuple_list() <scrapy.http.headers.Headers.to_tuple_list>`
+ that returns headers as a list of ``(key, value)`` tuples.
+ (:issue:`7239`)
+
+ .. _release-2.14.2:
+
+ Scrapy 2.14.2 (2026-03-12)
+ --------------------------
+
+ Security bug fixes
+ ~~~~~~~~~~~~~~~~~~
+
+ - Values from the ``Referrer-Policy`` header of HTTP responses are no longer
+ executed as Python callables. See the `cwxj-rr6w-m6w7`_ security advisory
+ for details.
+
+ .. _cwxj-rr6w-m6w7: https://github.com/scrapy/scrapy/security/advisories/GHSA-cwxj-rr6w-m6w7
+
+ - In line with the `standard
+ <https://fetch.spec.whatwg.org/#http-redirect-fetch>`__, 301 redirects of
+ ``POST`` requests are converted into ``GET`` requests.
+
+ Converting to a ``GET`` request implies not only a method change, but also
+ omitting the body and ``Content-*`` headers in the redirect request. On
+ cross-origin redirects (for example, cross-domain redirects), this is
+ effectively a security bug fix for scenarios where the body contains
+ secrets.
+
+ Deprecations
+ ~~~~~~~~~~~~
+
+ - Passing a response URL string as the first positional argument to
+ :meth:`scrapy.spidermiddlewares.referer.RefererMiddleware.policy` is
+ deprecated. Pass a :class:`~scrapy.http.Response` instead.
+
+ The parameter has also been renamed to ``response`` to reflect this change.
+ The old parameter name (``resp_or_url``) is deprecated.
+
+ New features
+ ~~~~~~~~~~~~
+
+ - Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing
+ supported referrer policies.
+
+ Bug fixes
+ ~~~~~~~~~
+
+ - Made additional redirect scenarios convert to ``GET`` in line with the
+ `standard <https://fetch.spec.whatwg.org/#http-redirect-fetch>`__:
+
+ - Only ``POST`` 302 redirects are converted into ``GET`` requests; other
+ methods are preserved.
+
+ - ``HEAD`` 303 redirects are not converted into ``GET`` requests.
+
+ - ``GET`` 303 redirects do not have their body or standard ``Content-*``
+ headers removed.
+
+ - Redirects where the original request body is dropped now also have their
+ ``Content-Encoding``, ``Content-Language`` and ``Content-Location`` headers
+ removed, in addition to the ``Content-Type`` and ``Content-Length`` headers
+ that were already being removed.
+
+ - Redirects now preserve the source URL fragment if the redirect URL does not
+ include one. This is useful when using browser-based download handlers,
+ such as `scrapy-playwright`_ or `scrapy-zyte-api`_, while letting Scrapy
+ handle redirects.
+
+ .. _scrapy-playwright: https://github.com/scrapy-plugins/scrapy-playwright
+ .. _scrapy-zyte-api: https://scrapy-zyte-api.readthedocs.io/en/latest/
+
+ - The ``Referer`` header is now removed on redirect if
+ :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` is disabled.
+
+ - The handling of the ``Referer`` header on redirects now takes into account
+ the ``Referer-Policy`` header of the response that triggers the redirect.
+
.. _release-2.14.1:
Scrapy 2.14.1 (2026-01-12)
diff --cc pyproject.toml
index 5138fda88,98dd0b42c..28e297401
--- a/pyproject.toml
+++ b/pyproject.toml
@@@ -123,39 -106,12 +123,39 @@@ implicit_reexport = tru
module = "scrapy.settings.default_settings"
ignore_errors = true
+# deprecated modules
[[tool.mypy.overrides]]
-module = "twisted"
-implicit_reexport = true
+module = [
+ "scrapy.core.downloader.webclient",
+ "scrapy.spiders.init",
+ "scrapy.utils.testsite",
+ "tests.test_webclient",
+]
+allow_any_generics = true
+allow_untyped_calls = true
+allow_untyped_defs = true
+check_untyped_defs = false
+warn_return_any = false
+
+# usually no type hints
+[[tool.mypy.overrides]]
+module = [
+# "IPython.*",
+ "bpython",
+ "brotli",
+ "brotlicffi",
+ "google.*",
+ "pydispatch.*",
+ "pyftpdlib.*",
+ "pytest_twisted",
+ "robotexclusionrulesparser",
+ "testfixtures",
+ "zope.interface.*",
+]
+ignore_missing_imports = true
[tool.bumpversion]
- current_version = "2.14.1"
+ current_version = "2.14.2"
commit = true
tag = true
tag_name = "{new_version}"