Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Release notes for 2.13.0, up to b4c253102139e842859a9abf1455e62504cc9511.

Andrey Rakhmatullinb1f85b5a173f
diff --git a/docs/news.rst b/docs/news.rst
index 9f476ee21..b9b5ce320 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -5,21 +5,209 @@ Release notes
.. _release-VERSION:
-Scrapy VERSION (unreleased)
----------------------------
+Scrapy 2.13.0 (unreleased)
+--------------------------
+
+Highlights:
+
+- Added the :reqmeta:`allow_offsite` request meta key
+
+- HTTP/1.0 support is deprecated
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Dropped support for PyPy 3.9.
+ (:issue:`6613`)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The ``from_settings()`` method of
- :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is removed
- without a deprecation period (this was needed because after the
- introduction of the
+ :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
+ deprecated in 2.12.0, is removed earlier than the usual deprecation period
+ (this was needed because after the introduction of the
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
switching built-in spider middlewares to it those middlewares need the
:class:`~scrapy.crawler.Crawler` instance at run time). Please use
``from_crawler()`` instead.
+Deprecations
+~~~~~~~~~~~~
+
+- Functions that were imported from :mod:`w3lib.url` and re-exported in
+ :mod:`scrapy.utils.url` are now deprecated, you should import them from
+ ``w3lib.url`` directly. They are:
+
+ - ``scrapy.utils.url.add_or_replace_parameter()``
+
+ - ``scrapy.utils.url.add_or_replace_parameters()``
+
+ - ``scrapy.utils.url.any_to_uri()``
+
+ - ``scrapy.utils.url.canonicalize_url()``
+
+ - ``scrapy.utils.url.file_uri_to_path()``
+
+ - ``scrapy.utils.url.is_url()``
+
+ - ``scrapy.utils.url.parse_data_uri()``
+
+ - ``scrapy.utils.url.parse_url()``
+
+ - ``scrapy.utils.url.path_to_file_uri()``
+
+ - ``scrapy.utils.url.safe_download_url()``
+
+ - ``scrapy.utils.url.safe_url_string()``
+
+ - ``scrapy.utils.url.url_query_cleaner()``
+
+ - ``scrapy.utils.url.url_query_parameter()``
+
+ - ``scrapy.utils.url._unquotepath()``
+
+ - ``scrapy.utils.url._safe_chars`` attribute
+
+ (:issue:`4577`, :issue:`6583`, :issue:`6586`)
+
+- HTTP/1.0 support code is deprecated. It was disabled by default and
+ couldn't be used together with HTTP/1.1. If you still need it, you should
+ write your own download handler or copy the code from Scrapy. The
+ deprecations include:
+
+ - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter``
+
+ - Overriding
+ ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()``
+
+ (:issue:`6634`)
+
+- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
+ :func:`scrapy.utils.versions.get_versions()` instead.
+ (:issue:`6582`)
+
+- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be
+ called.
+ (:issue:`4151`)
+
+New features
+~~~~~~~~~~~~
+
+- Added the :reqmeta:`allow_offsite` request meta key that can be used
+ instead of the more general :attr:`~scrapy.Request.dont_filter` request
+ attribute to skip processing of the request by
+ :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not
+ by other code that checks :attr:`~scrapy.Request.dont_filter`).
+ (:issue:`3690`, :issue:`6151`, :issue:`6366`)
+
+- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called
+ ``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
+ <pre-crawler-settings>`.
+ (:issue:`6544`, :issue:`6568`)
+
+- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
+ :attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
+ customizing the log level of the message that is logged when an item is
+ dropped.
+ (:issue:`6603`, :issue:`6608`)
+
+- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
+ list of software which versions are logged when the spider starts.
+ (:issue:`6582`)
+
+Improvements
+~~~~~~~~~~~~
+
+- Improved the error message when running a ``scrapy`` command that requires
+ a project (such as ``scrapy crawl``) outside of a project directory.
+ (:issue:`2349`, :issue:`3426`)
+
+- An empty :setting:`ADDONS` setting added to the ``settings.py`` template
+ for new projects.
+ (:issue:`6587`)
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed calculation of ``items_per_minute`` and ``responses_per_minute``
+ stats.
+ (:issue:`6599`)
+
+- Fixed an error initializing
+ :class:`scrapy.extensions.feedexport.GCSFeedStorage`.
+ (:issue:`6617`, :issue:`6628`)
+
+- Fixed an error running ``scrapy bench``.
+ (:issue:`6632`, :issue:`6633`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Improved the contribution docs.
+ (:issue:`6561`, :issue:`6575`)
+
+- Other documentation improvements and fixes.
+ (:issue:`4151`,
+ :issue:`6526`,
+ :issue:`6620`,
+ :issue:`6621`,
+ :issue:`6622`,
+ :issue:`6623`,
+ :issue:`6624`)
+
+Packaging
+~~~~~~~~~
+
+- Switched from ``setup.py`` to ``pyproject.toml``.
+ (:issue:`6514`, :issue:`6547`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Replaced most linters with ``ruff``.
+ (:issue:`6565`, :issue:`6576`, :issue:`6577`, :issue:`6581`, :issue:`6584`,
+ :issue:`6595`, :issue:`6601`, :issue:`6631`)
+
+- Improved accuracy and performance of collecting test coverage.
+ (:issue:`6567`)
+
+- Fixed an error that prevented running tests from directories other than the
+ top level source directory.
+ (:issue:`6567`)
+
+- Reduced the amount of ``mockserver`` calls in tests to improve the overall
+ test run time.
+ (:issue:`6637`, :issue:`6648`)
+
+- Fixed tests that were running the same test code more than once.
+ (:issue:`6646`)
+
+- Type hints improvements and fixes.
+ (:issue:`6578`, :issue:`6579`, :issue:`6593`, :issue:`6605`)
+
+- CI and test improvements and fixes.
+ (:issue:`5360`,
+ :issue:`6271`,
+ :issue:`6547`,
+ :issue:`6560`,
+ :issue:`6602`,
+ :issue:`6607`,
+ :issue:`6609`,
+ :issue:`6613`,
+ :issue:`6619`,
+ :issue:`6626`)
+
+- Code cleanups.
+ (:issue:`6600`,
+ :issue:`6606`,
+ :issue:`6635`)
+
+
.. _release-2.12.0:
Scrapy 2.12.0 (2024-11-18)