Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge remote-tracking branch 'origin/master' into relnotes-2.13.0

Andrey Rakhmatullinb07e6d4ea886
diff --cc docs/news.rst
index d9bc572be,64a3ad2b1..d574317bb
--- a/docs/news.rst
+++ b/docs/news.rst
@@@ -10,421 -10,130 +10,532 @@@ Scrapy 2.13.0 (unreleased
Highlights:
+- The asyncio reactor is now enabled by default
+
+ - Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`
+ (async) and changed how it is iterated.
+
+- Added the :reqmeta:`allow_offsite` request meta key
+
+- :ref:`Spider middlewares that don't support asynchronous spider output
+ <sync-async-spider-middleware>` are deprecated
+
+- Added a base class for :ref:`universal spider middlewares
+ <universal-spider-middleware>`
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Dropped support for PyPy 3.9.
+ (:issue:`6613`)
+
+- Added support for PyPy 3.11.
+ (:issue:`6697`)
+
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- The ``from_settings()`` method of
- :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is removed
- without a deprecation period (this was needed because after the
- introduction of the
- :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
- switching built-in spider middlewares to it those middlewares need the
- :class:`~scrapy.crawler.Crawler` instance at run time). Please use
- ``from_crawler()`` instead.
+- The default value of the :setting:`TWISTED_REACTOR` setting was changed
+ from ``None`` to
+ ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
+ was used in newly generated projects since Scrapy 2.7.0 but now existing
+ projects that don't explicitly set this setting will also use the asyncio
+ reactor. You can :ref:`change this setting in your project
+ <disable-asyncio>` to use a different reactor.
+ (:issue:`6659`, :issue:`6713`)
+ - The iteration of start requests and items no longer stops once there are
+ requests in the scheduler, and instead runs continuously until all start
+ requests have been scheduled.
+
+ To reproduce the previous behavior, see :ref:`start-requests-lazy`.
++ (:issue:`6729`)
+
+ - An unhandled exception from the
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.open_spider` method of a
+ :ref:`spider middleware <topics-spider-middleware>` no longer stops the
+ crawl.
++ (:issue:`6729`)
+
+ - In ``scrapy.core.engine.ExecutionEngine``:
+
+ - The second parameter of ``open_spider()``, ``start_requests``, has been
+ removed. The start requests are determined by the ``spider`` parameter
+ instead (see :meth:`~scrapy.Spider.start`).
+
+ - The ``slot`` attribute has been renamed to ``_slot`` and should not be
+ used.
+
++ (:issue:`6729`)
++
+ - In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot``
+ and should not be used.
++ (:issue:`6729`)
+
+ - The ``slot`` :ref:`telnet variable <telnet-vars>` has been removed.
++ (:issue:`6729`)
+
+ - In ``scrapy.core.spidermw.SpiderMiddlewareManager``,
+ ``process_start_requests()`` has been replaced by ``process_start()``.
++ (:issue:`6729`)
+
+ - The now-deprecated ``start_requests()`` method, when it returns an iterable
+ instead of being defined as a generator, is now executed *after* the
+ :ref:`scheduler <topics-scheduler>` instance has been created.
++ (:issue:`6729`)
+
+ - When using :setting:`JOBDIR`, :ref:`start requests <start-requests>` are
+ now serialized into their own, ``s``-suffixed priority folders. You can set
+ :setting:`SCHEDULER_START_DISK_QUEUE` to ``None`` or ``""`` to change that,
+ but the side effects may be undesirable. See
+ :setting:`SCHEDULER_START_DISK_QUEUE` for details.
++ (:issue:`6729`)
++
+- The URL length limit, set by the :setting:`URLLENGTH_LIMIT` setting, is now
+ also enforced for start requests.
+ (:issue:`6777`)
+
+- The ``from_settings()`` method of
+ :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
+ deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation
+ period (this was needed because after the introduction of the
+ :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
+ switching built-in spider middlewares to it those middlewares need the
+ :class:`~scrapy.crawler.Crawler` instance at run time). Please use
+ ``from_crawler()`` instead.
+ (:issue:`6693`)
+
+- ``scrapy.utils.url.escape_ajax()`` is no longer called when a
+ :class:`~scrapy.Request` instance is created. It was only useful for
+ websites supporting the ``_escaped_fragment_`` feature which most modern
+ websites don't support. If you still need this you can modify the URLs
+ before passing them to :class:`~scrapy.Request`.
+ (:issue:`6523`, :issue:`6651`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- Removed old deprecated name aliases for some signals:
+
+ - ``stats_spider_opened`` (use ``spider_opened`` instead)
+
+ - ``stats_spider_closing`` and ``stats_spider_closed`` (use
+ ``spider_closed`` instead)
+
+ - ``item_passed`` (use ``item_scraped`` instead)
+
+ - ``request_received`` (use ``request_scheduled`` instead)
+
+ (:issue:`6654`, :issue:`6655`)
Deprecations
~~~~~~~~~~~~
+ - The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated,
+ use :meth:`~scrapy.Spider.start` instead, or both to maintain support for
+ lower Scrapy versions.
-
+ (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`)
+
+ - The ``process_start_requests()`` method of :ref:`spider middlewares
+ <topics-spider-middleware>` is deprecated, use
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
+ or both to maintain support for lower Scrapy versions.
-
+ (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`)
+
+ - The ``__init__`` method of priority queue classes (see
+ :setting:`SCHEDULER_PRIORITY_QUEUE`) should now support a keyword-only
+ ``start_queue_cls`` parameter.
-
+ (:issue:`6752`)
+
+- :ref:`Spider middlewares that don't support asynchronous spider output
+ <sync-async-spider-middleware>` are deprecated. The async iterable
+ downgrading feature, needed for using such middlewares with asynchronous
+ callbacks and with other spider middlewares that produce asynchronous
+ iterables, is also deprecated. Please update all such middlewares to
+ support asynchronous spider output.
+ (:issue:`6664`)
+
+- Functions that were imported from :mod:`w3lib.url` and re-exported in
+ :mod:`scrapy.utils.url` are now deprecated, you should import them from
+ :mod:`w3lib.url` directly. They are:
+
+ - ``scrapy.utils.url.add_or_replace_parameter()``
+
+ - ``scrapy.utils.url.add_or_replace_parameters()``
+
+ - ``scrapy.utils.url.any_to_uri()``
+
+ - ``scrapy.utils.url.canonicalize_url()``
+
+ - ``scrapy.utils.url.file_uri_to_path()``
+
+ - ``scrapy.utils.url.is_url()``
+
+ - ``scrapy.utils.url.parse_data_uri()``
+
+ - ``scrapy.utils.url.parse_url()``
+
+ - ``scrapy.utils.url.path_to_file_uri()``
+
+ - ``scrapy.utils.url.safe_download_url()``
+
+ - ``scrapy.utils.url.safe_url_string()``
+
+ - ``scrapy.utils.url.url_query_cleaner()``
+
+ - ``scrapy.utils.url.url_query_parameter()``
+
+ (:issue:`4577`, :issue:`6583`, :issue:`6586`)
+
+- HTTP/1.0 support code is deprecated. It was disabled by default and
+ couldn't be used together with HTTP/1.1. If you still need it, you should
+ write your own download handler or copy the code from Scrapy. The
+ deprecations include:
+
+ - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter``
+
+ - Overriding
+ ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()``
+
+ (:issue:`6634`)
+
+- The following modules and functions used only in tests are deprecated:
+
+ - the ``scrapy/utils/testproc`` module
+
+ - the ``scrapy/utils/testsite`` module
+
+ - ``scrapy.utils.test.assert_gcs_environ()``
+
+ - ``scrapy.utils.test.get_ftp_content_and_delete()``
+
+ - ``scrapy.utils.test.get_gcs_content_and_delete()``
+
+ - ``scrapy.utils.test.mock_google_cloud_storage()``
+
+ - ``scrapy.utils.test.skip_if_no_boto()``
+
+ If you need to use them in your tests or code, you can copy the code from Scrapy.
+ (:issue:`6696`)
+
+- ``scrapy.utils.test.TestSpider`` is deprecated. If you need an empty spider
+ class you can use :class:`scrapy.utils.spider.DefaultSpider` or create your
+ own subclass of :class:`scrapy.Spider`.
+ (:issue:`6678`)
+
+- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is
+ deprecated. It was disabled by default and isn't useful for most of the
+ existing websites.
+ (:issue:`6523`, :issue:`6651`, :issue:`6656`)
+
+- ``scrapy.utils.url.escape_ajax()`` is deprecated.
+ (:issue:`6523`, :issue:`6651`)
+
+- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,
+ you can copy its code from Scrapy.
+ (:issue:`6708`, :issue:`6714`)
+
+- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
+ :func:`scrapy.utils.versions.get_versions()` instead.
+ (:issue:`6582`)
+
+- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be
+ called.
+ (:issue:`4151`)
+
+- Passing the ``spider`` argument to the following methods of
+ :class:`~scrapy.core.scraper.Scraper` is deprecated:
+
+ - ``close_spider()``
+
+ - ``enqueue_scrape()``
+
+ - ``handle_spider_error()``
+
+ - ``handle_spider_output()``
+
+ (:issue:`6764`)
+
New features
~~~~~~~~~~~~
+ - You can now yield the start requests and items of a spider from the
+ :meth:`~scrapy.Spider.start` spider method and from the
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` spider
+ middleware method, both :term:`asynchronous generators <python:asynchronous
+ generator>`.
+
+ This makes it possible to use asynchronous code to generate those start
+ requests and items, e.g. reading them from a queue service or database
+ using an asynchronous client, without workarounds.
-
+ (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`)
+
+ - Start requests are now :ref:`scheduled <topics-scheduler>` as soon as
+ possible.
+
+ As a result, their :attr:`~scrapy.Request.priority` is now taken into
+ account as soon as :setting:`CONCURRENT_REQUESTS` is reached.
-
+ (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`, :issue:`6729`)
+
+ - :class:`Crawler.signals <scrapy.signalmanager.SignalManager>` has a new
+ :meth:`~scrapy.signalmanager.SignalManager.wait_for` method.
++ (:issue:`6729`)
+
+ - Added a new :signal:`scheduler_empty` signal.
++ (:issue:`6729`)
+
+ - Added new settings: :setting:`SCHEDULER_START_DISK_QUEUE` and
+ :setting:`SCHEDULER_START_MEMORY_QUEUE`.
++ (:issue:`6729`)
+
+ - Added :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware`, which
+ sets :reqmeta:`is_start_request` to ``True`` on :ref:`start requests
+ <start-requests>`.
++ (:issue:`6729`)
+
+ - Exposed a new method of :class:`Crawler.engine
+ <scrapy.core.engine.ExecutionEngine>`:
+ :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`.
++ (:issue:`6729`)
++
+- Added the :reqmeta:`allow_offsite` request meta key that can be used
+ instead of the more general :attr:`~scrapy.Request.dont_filter` request
+ attribute to skip processing of the request by
+ :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not
+ by other code that checks :attr:`~scrapy.Request.dont_filter`).
+ (:issue:`3690`, :issue:`6151`, :issue:`6366`)
+
+- Added an optional base class for spider middlewares,
+ :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be
+ helpful for writing :ref:`universal spider middlewares
+ <universal-spider-middleware>` without boilerplate and code duplication.
+ The built-in spider middlewares now inherit from this class.
+ (:issue:`6693`, :issue:`6777`)
+
+- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called
+ ``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
+ <pre-crawler-settings>`.
+ (:issue:`6544`, :issue:`6568`)
+
+- Added :ref:`helpers <priority-dict-helpers>` for modifying :ref:`component
+ priority dictionary <component-priority-dictionaries>` settings.
+ (:issue:`6614`)
+
+- Responses that use an unknown/unsupported encoding now produce a warning.
+ If Scrapy knows that installing an additional package (such as brotli_)
+ will allow decoding the response, that will be mentioned in the warning.
+ (:issue:`4697`, :issue:`6618`)
+
+- Added the ``spider_exceptions/count`` stat which tracks the total count of
+ exceptions (tracked also by per-type ``spider_exceptions/*`` stats).
+ (:issue:`6739`, :issue:`6740`)
+
+- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
+ :attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
+ customizing the log level of the message that is logged when an item is
+ dropped.
+ (:issue:`6603`, :issue:`6608`)
+
+- Added support for the ``-b, --cookie`` curl argument to
+ :meth:`scrapy.Request.from_curl`.
+ (:issue:`6684`)
+
+- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
+ list of software whose versions are logged when the spider starts.
+ (:issue:`6582`)
+
+- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows
+ disabling run time analysis of callback code used to warn about incorrect
+ ``return`` statements in generator-based callbacks. You may need to disable
+ this setting if this analysis breaks on your callback code.
+ (:issue:`6731`, :issue:`6738`)
+
+Improvements
+~~~~~~~~~~~~
+
+- Removed or postponed some calls of :func:`itemadapter.is_item` to increase
+ performance.
+ (:issue:`6719`)
+
+- Improved the error message when running a ``scrapy`` command that requires
+ a project (such as ``scrapy crawl``) outside of a project directory.
+ (:issue:`2349`, :issue:`3426`)
+
+- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template
+ for new projects.
+ (:issue:`6587`)
Bug fixes
~~~~~~~~~
+ - Yielding an item from :meth:`Spider.start <scrapy.Spider.start>` or from
+ :meth:`SpiderMiddleware.process_start
+ <scrapy.spidermiddlewares.SpiderMiddleware.process_start>` no longer delays
+ the next iteration of starting requests and items by up to 5 seconds.
-
+ (:issue:`6729`)
+
+- Fixed calculation of ``items_per_minute`` and ``responses_per_minute``
+ stats.
+ (:issue:`6599`)
+
+- Fixed an error initializing
+ :class:`scrapy.extensions.feedexport.GCSFeedStorage`.
+ (:issue:`6617`, :issue:`6628`)
+
+- Fixed an error running ``scrapy bench``.
+ (:issue:`6632`, :issue:`6633`)
+
+- Fixed duplicated log messages about the reactor and the event loop.
+ (:issue:`6636`, :issue:`6657`)
+
+- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at
+ run time, required by tools such as scrapy-poet_.
+ (:issue:`6665`, :issue:`6671`)
+
+ .. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet
+
+- Calling :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` without
+ an installed reactor now raises an exception instead of installing a
+ reactor.
+ (:issue:`6732`, :issue:`6735`)
+
+- Restored support for the ``x-gzip`` content encoding.
+ (:issue:`6618`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Documented the setting values set in the default project template.
+ (:issue:`6762`, :issue:`6775`)
+
+- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous
+ iterable support in spider middlewares.
+ (:issue:`6688`)
+
+- Improved the :ref:`docs <coroutine-deferred-apis>` about using
+ :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
+ code and included a list of such APIs.
+ (:issue:`6677`, :issue:`6734`, :issue:`6776`)
+
+- Improved the :ref:`contribution docs <topics-contributing>`.
+ (:issue:`6561`, :issue:`6575`)
+
+- Removed the ``Splash`` recommendation from the :ref:`headless browser
+ <topics-headless-browsing>` suggestion. We no longer recommend using
+ ``Splash`` and recommend using other headless browser solutions instead.
+ (:issue:`6642`, :issue:`6701`)
+
+- Added the dark mode to the HTML documentation.
+ (:issue:`6653`)
+
+- Other documentation improvements and fixes.
+ (:issue:`4151`,
+ :issue:`6526`,
+ :issue:`6620`,
+ :issue:`6621`,
+ :issue:`6622`,
+ :issue:`6623`,
+ :issue:`6624`,
+ :issue:`6721`,
+ :issue:`6723`,
+ :issue:`6780`)
+
+Packaging
+~~~~~~~~~
+
+- Switched from ``setup.py`` to ``pyproject.toml``.
+ (:issue:`6514`, :issue:`6547`)
+
+- Switched the build backend from setuptools_ to hatchling_.
+ (:issue:`6771`)
+
+ .. _hatchling: https://pypi.org/project/hatchling/
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Replaced most linters with ruff_.
+ (:issue:`6565`,
+ :issue:`6576`,
+ :issue:`6577`,
+ :issue:`6581`,
+ :issue:`6584`,
+ :issue:`6595`,
+ :issue:`6601`,
+ :issue:`6631`)
+
+ .. _ruff: https://docs.astral.sh/ruff/
+
+- Improved accuracy and performance of collecting test coverage.
+ (:issue:`6255`, :issue:`6610`)
+
+- Fixed an error that prevented running tests from directories other than the
+ top level source directory.
+ (:issue:`6567`)
+
+- Reduced the amount of ``mockserver`` calls in tests to improve the overall
+ test run time.
+ (:issue:`6637`, :issue:`6648`)
+
+- Fixed tests that were running the same test code more than once.
+ (:issue:`6646`, :issue:`6647`, :issue:`6650`)
+
+- Refactored tests to use more ``pytest`` features instead of ``unittest``
+ ones where possible.
+ (:issue:`6678`,
+ :issue:`6680`,
+ :issue:`6695`,
+ :issue:`6699`,
+ :issue:`6700`,
+ :issue:`6702`,
+ :issue:`6709`,
+ :issue:`6710`,
+ :issue:`6711`,
+ :issue:`6712`,
+ :issue:`6725`)
+
+- Type hints improvements and fixes.
+ (:issue:`6578`,
+ :issue:`6579`,
+ :issue:`6593`,
+ :issue:`6605`,
+ :issue:`6694`)
+
+- CI and test improvements and fixes.
+ (:issue:`5360`,
+ :issue:`6271`,
+ :issue:`6547`,
+ :issue:`6560`,
+ :issue:`6602`,
+ :issue:`6607`,
+ :issue:`6609`,
+ :issue:`6613`,
+ :issue:`6619`,
+ :issue:`6626`,
+ :issue:`6679`,
+ :issue:`6703`,
+ :issue:`6704`,
+ :issue:`6716`,
+ :issue:`6720`,
+ :issue:`6722`,
+ :issue:`6724`,
+ :issue:`6741`,
+ :issue:`6743`,
+ :issue:`6766`,
+ :issue:`6770`,
+ :issue:`6772`,
+ :issue:`6773`)
+
+- Code cleanups.
+ (:issue:`6600`,
+ :issue:`6606`,
+ :issue:`6635`,
+ :issue:`6764`)
+
.. _release-2.12.0:
diff --cc scrapy/spidermiddlewares/base.py
index 5e4370d45,cfb50c599..196b84d0d
--- a/scrapy/spidermiddlewares/base.py
+++ b/scrapy/spidermiddlewares/base.py
@@@ -17,11 -17,9 +17,11 @@@ if TYPE_CHECKING
class BaseSpiderMiddleware:
"""Optional base class for spider middlewares.
+ .. versionadded:: 2.13
+
- This class provides helper methods for asynchronous ``process_spider_output``
- methods. Middlewares that don't have a ``process_spider_output`` method don't need
- to use it.
+ This class provides helper methods for asynchronous
+ ``process_spider_output()`` and ``process_start()`` methods. Middlewares
+ that don't have either of these methods don't need to use this class.
You can override the
:meth:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware.get_processed_request`