Scrapy, a fast high-level web crawling & scraping framework for Python.
Cover the current master in the release notes (up to daf9db7).
Andrey Rakhmatullin095140f13474
diff --git a/docs/news.rst b/docs/news.rstindex b9b5ce320..74a26c383 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,16 +3,22 @@Release notes=============-.. _release-VERSION:+.. _release-2.13.0:Scrapy 2.13.0 (unreleased)--------------------------Highlights:+- The asyncio reactor is now enabled by default+- Added the :reqmeta:`allow_offsite` request meta key-- HTTP/1.0 support is deprecated+- :ref:`Spider middlewares that don't support asynchronous spider output+ <sync-async-spider-middleware>` are deprecated++- Added a base class for :ref:`universal spider middlewares+ <universal-spider-middleware>`Modified requirements~~~~~~~~~~~~~~~~~~~~~@@ -23,21 +29,62 @@ Modified requirementsBackward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+- The default value of the :setting:`TWISTED_REACTOR` setting was changed+ from ``None`` to+ ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value+ was used in newly generated projects since Scrapy 2.7.0 but now existing+ projects that don't explicitly set this setting will also use the asyncio+ reactor. You can :ref:`change this setting in your project+ <disable-asyncio>` to use a different reactor.+ (:issue:`6659`, :issue:`6713`)+- The ``from_settings()`` method of:class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,- deprecated in 2.12.0, is removed earlier than the usual deprecation period- (this was needed because after the introduction of the+ deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation+ period (this was needed because after the introduction of the:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class andswitching built-in spider middlewares to it those middlewares need the:class:`~scrapy.crawler.Crawler` instance at run time). Please use``from_crawler()`` instead.+ (:issue:`6693`)++- ``scrapy.utils.url.escape_ajax()`` is no longer called when a+ :class:`~scrapy.Request` instance is created. It was only useful for+ websites supporting the ``_escaped_fragment_`` feature which most modern+ websites don't support. If you still need this you can modify the URLs+ before passing them to :class:`~scrapy.Request`.+ (:issue:`6523`, :issue:`6651`)++Deprecation removals+~~~~~~~~~~~~~~~~~~~~++- Removed old deprecated name aliases for some signals:++ - ``stats_spider_opened`` (use ``spider_opened`` instead)++ - ``stats_spider_closing`` and ``stats_spider_closed`` (use+ ``spider_closed`` instead)++ - ``item_passed`` (use ``item_scraped`` instead)++ - ``request_received`` (use ``request_scheduled`` instead)++ (:issue:`6654`, :issue:`6655`)Deprecations~~~~~~~~~~~~+- :ref:`Spider middlewares that don't support asynchronous spider output+ <sync-async-spider-middleware>` are deprecated. The async iterable+ downgrading feature, needed for using such middlewares with asynchronous+ callbacks and with other spider middlewares that produce asynchronous+ iterables, is also deprecated. Please update all such middlewares to+ support asynchronous spider output.+ (:issue:`6664`)+- Functions that were imported from :mod:`w3lib.url` and re-exported in:mod:`scrapy.utils.url` are now deprecated, you should import them from- ``w3lib.url`` directly. They are:+ :mod:`w3lib.url` directly. They are:- ``scrapy.utils.url.add_or_replace_parameter()``@@ -65,10 +112,6 @@ Deprecations- ``scrapy.utils.url.url_query_parameter()``- - ``scrapy.utils.url._unquotepath()``-- - ``scrapy.utils.url._safe_chars`` attribute-(:issue:`4577`, :issue:`6583`, :issue:`6586`)- HTTP/1.0 support code is deprecated. It was disabled by default and@@ -87,6 +130,37 @@ Deprecations(:issue:`6634`)+- The following modules and functions used only in tests are deprecated:++ - the ``scrapy/utils/testproc`` module++ - the ``scrapy/utils/testsite`` module++ - ``scrapy.utils.test.assert_gcs_environ()``++ - ``scrapy.utils.test.get_ftp_content_and_delete()``++ - ``scrapy.utils.test.get_gcs_content_and_delete()``++ - ``scrapy.utils.test.mock_google_cloud_storage()``++ - ``scrapy.utils.test.skip_if_no_boto()``++ If you need to use them in your tests or code, you can copy the code from Scrapy.+ (:issue:`6696`)++- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is+ deprecated. It was disabled by default and isn't useful for most of the+ existing websites.+ (:issue:`6523`, :issue:`6651`, :issue:`6656`)++- ``scrapy.utils.url.escape_ajax()`` is deprecated.+ (:issue:`6523`, :issue:`6651`)++- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,+ you can copy its code from Scrapy.+ (:issue:`6708`, :issue:`6714`)+- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use:func:`scrapy.utils.versions.get_versions()` instead.(:issue:`6582`)@@ -105,29 +179,63 @@ New featuresby other code that checks :attr:`~scrapy.Request.dont_filter`).(:issue:`3690`, :issue:`6151`, :issue:`6366`)+- Added an optional base class for spider middlewares,+ :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be+ helpful for writing :ref:`universal spider middlewares+ <universal-spider-middleware>` without boilerplate and code duplication.+ The built-in spider middlewares now inherit from this class.+ (:issue:`6693`)+- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings<pre-crawler-settings>`.(:issue:`6544`, :issue:`6568`)+- Added :ref:`helpers <priority-dict-helpers>` for modifying :ref:`component+ priority dictionary <component-priority-dictionaries>` settings.+ (:issue:`6614`)++- Responses that use an unknown/unsupported encoding now produce a warning.+ If Scrapy knows that installing an additional package (such as brotli_)+ will allow decoding the response, that will be mentioned in the warning.+ (:issue:`4697`, :issue:`6618`)++- Added the ``spider_exceptions/count`` stat which tracks the total count of+ exceptions (tracked also by per-type ``spider_exceptions/*`` stats).+ (:issue:`6739`, :issue:`6740`)+- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the:attr:`scrapy.exceptions.DropItem.log_level` attribute that allowcustomizing the log level of the message that is logged when an item isdropped.(:issue:`6603`, :issue:`6608`)+- Added support for the ``-b, --cookie`` curl argument to+ :meth:`scrapy.Request.from_curl`.+ (:issue:`6684`)+- Added the :setting:`LOG_VERSIONS` setting that allows customizing the- list of software which versions are logged when the spider starts.+ list of software whose versions are logged when the spider starts.(:issue:`6582`)+- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows+ disabling run time analysis of callback code used to warn about incorrect+ ``return`` statements in generator-based callbacks. You may need to disable+ this setting if this analysis breaks on your callback code.+ (:issue:`6731`, :issue:`6738`)+Improvements~~~~~~~~~~~~+- Removed or postponed some calls of :func:`itemadapter.is_item` to increase+ performance.+ (:issue:`6719`)+- Improved the error message when running a ``scrapy`` command that requiresa project (such as ``scrapy crawl``) outside of a project directory.(:issue:`2349`, :issue:`3426`)-- An empty :setting:`ADDONS` setting added to the ``settings.py`` template+- Added an empty :setting:`ADDONS` setting to the ``settings.py`` templatefor new projects.(:issue:`6587`)@@ -145,12 +253,46 @@ Bug fixes- Fixed an error running ``scrapy bench``.(:issue:`6632`, :issue:`6633`)+- Fixed duplicated log messages about the reactor and the event loop.+ (:issue:`6636`, :issue:`6657`)++- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at+ run time, required by tools such as scrapy-poet_.+ (:issue:`6665`, :issue:`6671`)++ .. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet++- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without+ an installed reactor now raises an exception instead of installing a+ reactor.+ (:issue:`6732`, :issue:`6735`)++- Restored support for the ``x-gzip`` content encoding.+ (:issue:`6618`)+Documentation~~~~~~~~~~~~~-- Improved the contribution docs.+- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous+ iterable support in spider middlewares.+ (:issue:`6688`)++- Improved the :ref:`docs <coroutine-deferred-apis>` about using+ :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based+ code.+ (:issue:`6734`)++- Improved the :ref:`contribution docs <topics-contributing>`.(:issue:`6561`, :issue:`6575`)+- Removed the ``Splash`` recommendation from the :ref:`headless browser+ <topics-headless-browsing>` suggestion. We no longer recommend using+ ``Splash`` and recommend using other headless browser solutions instead.+ (:issue:`6642`, :issue:`6701`)++- Added the dark mode to the HTML documentation.+ (:issue:`6653`)+- Other documentation improvements and fixes.(:issue:`4151`,:issue:`6526`,@@ -158,7 +300,9 @@ Documentation:issue:`6621`,:issue:`6622`,:issue:`6623`,- :issue:`6624`)+ :issue:`6624`,+ :issue:`6721`,+ :issue:`6723`)Packaging~~~~~~~~~@@ -169,12 +313,20 @@ PackagingQuality assurance~~~~~~~~~~~~~~~~~-- Replaced most linters with ``ruff``.- (:issue:`6565`, :issue:`6576`, :issue:`6577`, :issue:`6581`, :issue:`6584`,- :issue:`6595`, :issue:`6601`, :issue:`6631`)+- Replaced most linters with ruff_.+ (:issue:`6565`,+ :issue:`6576`,+ :issue:`6577`,+ :issue:`6581`,+ :issue:`6584`,+ :issue:`6595`,+ :issue:`6601`,+ :issue:`6631`)++ .. _ruff: https://docs.astral.sh/ruff/- Improved accuracy and performance of collecting test coverage.- (:issue:`6567`)+ (:issue:`6255`, :issue:`6610`)- Fixed an error that prevented running tests from directories other than thetop level source directory.@@ -185,10 +337,28 @@ Quality assurance(:issue:`6637`, :issue:`6648`)- Fixed tests that were running the same test code more than once.- (:issue:`6646`)+ (:issue:`6646`, :issue:`6647`, :issue:`6650`)++- Refactored tests to use more ``pytest`` features instead of ``unittest``+ ones where possible.+ (:issue:`6678`,+ :issue:`6680`,+ :issue:`6695`,+ :issue:`6699`,+ :issue:`6700`,+ :issue:`6702`,+ :issue:`6709`,+ :issue:`6710`,+ :issue:`6711`,+ :issue:`6712`,+ :issue:`6725`)- Type hints improvements and fixes.- (:issue:`6578`, :issue:`6579`, :issue:`6593`, :issue:`6605`)+ (:issue:`6578`,+ :issue:`6579`,+ :issue:`6593`,+ :issue:`6605`,+ :issue:`6694`)- CI and test improvements and fixes.(:issue:`5360`,@@ -200,7 +370,16 @@ Quality assurance:issue:`6609`,:issue:`6613`,:issue:`6619`,- :issue:`6626`)+ :issue:`6626`,+ :issue:`6679`,+ :issue:`6703`,+ :issue:`6704`,+ :issue:`6716`,+ :issue:`6720`,+ :issue:`6722`,+ :issue:`6724`,+ :issue:`6741`,+ :issue:`6743`)- Code cleanups.(:issue:`6600`,@@ -806,8 +985,6 @@ Bug fixes- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 infavor of brotli_. (:issue:`6261`)- .. _brotli: https://github.com/google/brotli-.. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotliinstead if you can.@@ -2391,8 +2568,6 @@ Scrapy 2.5.1 (2021-10-05)need to upgrade scrapy-splash to a greater version for it to continue towork.-.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash-.. _release-2.5.0:diff --git a/docs/topics/addons.rst b/docs/topics/addons.rstindex 17e3c177a..815501e66 100644--- a/docs/topics/addons.rst+++ b/docs/topics/addons.rst@@ -124,6 +124,8 @@ Set some basic configuration:"ITEM_PIPELINES", MyPipeline, 200)+.. _priority-dict-helpers:+.. tip:: When editing a :ref:`component priority dictionary<component-priority-dictionaries>` setting, like :setting:`ITEM_PIPELINES`,consider using setting methods likediff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 1c80857f6..1a84f893c 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -344,3 +344,9 @@ For example:feature will be removed, and all spider middlewares will be expectedto define their ``process_spider_output`` method as an asynchronousgenerator.++Since 2.13.0, Scrapy provides a base class,+:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements+the ``process_spider_output()`` and ``process_spider_output_async()`` methods,+so instead of duplicating the processing code you can override the+``get_processed_request()`` and/or the ``get_processed_item()`` method.diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex a59a61050..73ac36646 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -2005,7 +2005,7 @@ current platform.``twisted.internet.asyncioreactor.AsyncioSelectorReactor`` in the generated``settings.py`` file.-.. versionchanged:: VERSION+.. versionchanged:: 2.13The default value was changed from ``None`` to``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.diff --git a/scrapy/spidermiddlewares/base.py b/scrapy/spidermiddlewares/base.pyindex 650192095..5e4370d45 100644--- a/scrapy/spidermiddlewares/base.py+++ b/scrapy/spidermiddlewares/base.py@@ -17,6 +17,8 @@ if TYPE_CHECKING:class BaseSpiderMiddleware:"""Optional base class for spider middlewares.+ .. versionadded:: 2.13+This class provides helper methods for asynchronous ``process_spider_output``methods. Middlewares that don't have a ``process_spider_output`` method don't needto use it.