Scrapy, a fast high-level web crawling & scraping framework for Python.
Deprecate non-asyncio reactors (#8454)
Andrey Rakhmatullin00436ad5cfe6
diff --git a/conftest.py b/conftest.pyindex f7e18ee44..d113322fe 100644--- a/conftest.py+++ b/conftest.py@@ -201,6 +201,12 @@ def pytest_configure(config):if config.getoption("--reactor") == "none":install_reactor_import_hook()+ if config.getoption("--reactor") == "default":+ config.addinivalue_line(+ "filterwarnings",+ "ignore:Using a Twisted reactor other than:scrapy.exceptions.ScrapyDeprecationWarning",+ )+# Test modules made up mostly of tests that spawn a subprocess per test (see# tests/utils/cmdline.py). Collected in file order, they form one contiguousdiff --git a/docs/faq.rst b/docs/faq.rstindex 7bc8e879e..77660ebd0 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -388,8 +388,7 @@ How to deal with ``<class 'ValueError'>: filedescriptor out of range in select()This issue `has been reported`_ to appear when running broad crawls in macOS, where the defaultTwisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time.-If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch-to a different one in the same way.+If you are using this reactor explicitly, switch to the default one... _faq-stop-response-download:diff --git a/docs/news.rst b/docs/news.rstindex b19033f8b..9db7a6a19 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -2355,9 +2355,9 @@ Modified requirementsBackward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- If you set the :setting:`TWISTED_REACTOR` setting to a :ref:`non-asyncio- value <disable-asyncio>` at the :ref:`spider level <spider-settings>`, you- may now need to set the :setting:`FORCE_CRAWLER_PROCESS` setting to+- If you set the ``TWISTED_REACTOR`` setting to a non-asyncio+ value at the :ref:`spider level <spider-settings>`, you+ may now need to set the ``FORCE_CRAWLER_PROCESS`` setting to``True`` when running Scrapy via :ref:`its command-line tool<topics-commands-crawlerprocess>` to avoid a reactor mismatch exception.(:gh:`6845`)diff --git a/docs/news/2.x.rst b/docs/news/2.x.rstindex 6164b0cbc..21647f288 100644--- a/docs/news/2.x.rst+++ b/docs/news/2.x.rst@@ -205,13 +205,13 @@ Modified requirementsBackward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- The default value of the :setting:`TWISTED_REACTOR` setting was changed+- The default value of the ``TWISTED_REACTOR`` setting was changedfrom ``None`` to``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This valuewas used in newly generated projects since Scrapy 2.7.0 but now existingprojects that don't explicitly set this setting will also use the asyncio- reactor. You can :ref:`change this setting in your project- <disable-asyncio>` to use a different reactor.+ reactor. You can change this setting in your project to use a different+ reactor.(:gh:`6659`, :gh:`6713`)- The iteration of start requests and items no longer stops once there are@@ -2460,7 +2460,7 @@ Also fixes the following regressions introduced in 2.6.0:- Installing a Twisted reactor before Scrapy does (e.g. importing:mod:`twisted.internet.reactor` somewhere at the module level) no longerprevents Scrapy from starting, as long as a different reactor is not- specified in :setting:`TWISTED_REACTOR` (:gh:`5525`, :gh:`5528`)+ specified in ``TWISTED_REACTOR`` (:gh:`5525`, :gh:`5528`)- Fixed an exception that was being logged after the spider finished undercertain conditions (:gh:`5437`, :gh:`5440`)@@ -2739,7 +2739,7 @@ New featuresBug fixes~~~~~~~~~-- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings+- The ``TWISTED_REACTOR`` and :setting:`ASYNCIO_EVENT_LOOP` settingsare no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`.(:gh:`4485`, :gh:`5352`)@@ -3914,7 +3914,7 @@ Scrapy 2.0.1 (2020-03-18)* Removed top-level :mod:`~twisted.internet.reactor` imports to preventerrors about the wrong Twisted reactor being installed when setting a- different Twisted reactor using :setting:`TWISTED_REACTOR` (:gh:`4401`,+ different Twisted reactor using ``TWISTED_REACTOR`` (:gh:`4401`,:gh:`4406`)* Fixed tests (:gh:`4422`)@@ -4049,7 +4049,7 @@ New featuresexisting soft limit that pauses request downloads when the total responsedata being processed is too high (:gh:`1410`, :gh:`3551`)-* A new :setting:`TWISTED_REACTOR` setting allows customizing the+* A new ``TWISTED_REACTOR`` setting allows customizing the:mod:`~twisted.internet.reactor` that Scrapy uses, allowing to:doc:`enable asyncio support <topics/asyncio>` or deal with a:ref:`common macOS issue <faq-specific-reactor>` (:gh:`2905`,diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 643e274ec..3acc90c6e 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -4,10 +4,8 @@asyncio=======-Scrapy supports :mod:`asyncio` natively. New projects created with-:command:`startproject` have asyncio enabled by default, and you can use-:mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine-<coroutines>`.+Scrapy supports :mod:`asyncio` natively. You can use :mod:`asyncio` and+:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.The rest of this page covers advanced topics. If you are starting a new project,no additional setup is needed.@@ -15,28 +13,23 @@ no additional setup is needed... _install-asyncio:-Configuring the asyncio reactor-===============================+Installing the asyncio reactor+==============================-New projects generated with :command:`startproject` have the asyncio-reactor configured by default. No manual setup is needed.--The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy-uses. Its default value is-``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``, which enables-:mod:`asyncio` support.--If you are using :class:`~scrapy.crawler.AsyncCrawlerRunner` or-:class:`~scrapy.crawler.CrawlerRunner`, you also need to-install the :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`-reactor manually. You can do that using-:func:`~scrapy.utils.reactor.install_reactor`:+When Scrapy installs a reactor, it will install the correct one automatically.+If you are managing the reactor yourself, when using+:class:`~scrapy.crawler.AsyncCrawlerRunner` or+:class:`~scrapy.crawler.CrawlerRunner`, you need to install the+:class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` reactor+manually. You can do that using :func:`~scrapy.utils.reactor.install_reactor`:.. skip: next.. code-block:: pythoninstall_reactor()+.. autofunction:: scrapy.utils.reactor.install_reactor+.. _asyncio-preinstalled-reactor:@@ -47,8 +40,7 @@ Handling a pre-installed reactorTwisted reactor as a side effect. Once a Twisted reactor is installed, it isnot possible to switch to a different reactor at run time.-If you :ref:`configure the asyncio Twisted reactor <install-asyncio>` and, at-run time, Scrapy complains that a different reactor is already installed,+If at run time Scrapy complains that a different reactor is already installed,chances are you have some such imports in your code.You can usually fix the issue by moving those offending module-level Twisted@@ -84,21 +76,9 @@ Integrating Deferred code and asyncio code==========================================Coroutine functions can await on Deferreds by wrapping them into-:class:`asyncio.Future` objects. Scrapy provides two helpers for this:+:class:`asyncio.Future` objects. Scrapy provides a helper for this:.. autofunction:: scrapy.utils.defer.deferred_to_future-.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future--.. tip:: If you don't need to support reactors other than the default- :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`, you- can use :func:`~scrapy.utils.defer.deferred_to_future`, otherwise you- should use :func:`~scrapy.utils.defer.maybe_deferred_to_future`.--.. tip:: If you need to use these functions in code that aims to be compatible- with lower versions of Scrapy that do not provide these functions,- down to Scrapy 2.0 (earlier versions do not support- :mod:`asyncio`), you can copy the implementation of these functions- into your own code.Coroutines and futures can be wrapped into Deferreds (for example, when aScrapy API requires passing a Deferred to it) using the following helpers:@@ -110,33 +90,7 @@ The following function helps with a reverse wrapping:.. autofunction:: scrapy.utils.defer.ensure_awaitable--.. _enforce-asyncio-requirement:--Enforcing asyncio as a requirement-==================================--If you are writing a :ref:`component <topics-components>` that requires asyncio-to work, use :func:`scrapy.utils.asyncio.is_asyncio_available` to-:ref:`enforce it as a requirement <enforce-component-requirements>`. For-example:--.. code-block:: python-- from scrapy.utils.asyncio import is_asyncio_available--- class MyComponent:- def __init__(self):- if not is_asyncio_available():- raise ValueError(- f"{MyComponent.__qualname__} requires the asyncio support. "- f"Make sure you have configured the asyncio reactor in the "- f"TWISTED_REACTOR setting. See the asyncio documentation "- f"of Scrapy for more information."- )--.. autofunction:: scrapy.utils.asyncio.is_asyncio_available+.. seealso:: :ref:`asyncio-old-versions`.. _asyncio-without-reactor:@@ -248,8 +202,9 @@ Here are some examples of APIs and patterns that need a replacement:execute code in other threads. You can use :func:`asyncio.to_thread`,:meth:`asyncio.loop.call_soon_threadsafe` and related APIs instead.* Using :class:`twisted.internet.task.LoopingCall` for scheduling repeated- tasks. As there is no direct replacement in the standard library, you may- need to write your own one using :func:`asyncio.sleep` in a task.+ tasks. As there is no direct replacement in the standard library, you can use+ :class:`scrapy.utils.asyncio.AsyncioLoopingCall` or write your own simple+ implementation using :func:`asyncio.sleep` in a task.* Using Twisted network client and server APIs (:meth:`reactor.connectTCP()<twisted.internet.interfaces.IReactorTCP.connectTCP>`,:meth:`reactor.listenTCP()@@ -261,13 +216,10 @@ Here are some examples of APIs and patterns that need a replacement::class:`~scrapy.crawler.AsyncCrawlerProcess` or:class:`~scrapy.crawler.AsyncCrawlerRunner` respectively instead.-Scrapy provides unified helpers for some of these examples:+Scrapy provides this replacement for+:class:`~twisted.internet.task.LoopingCall`:-.. autofunction:: scrapy.utils.asyncio.sleep-.. autofunction:: scrapy.utils.asyncio.call_later-.. autofunction:: scrapy.utils.asyncio.create_looping_call.. autoclass:: scrapy.utils.asyncio.AsyncioLoopingCall-.. autofunction:: scrapy.utils.asyncio.run_in_threadIf your code needs to know whether the reactor is available, you can eithercheck for the value of the :setting:`TWISTED_REACTOR_ENABLED` setting (you need@@ -341,21 +293,85 @@ automatically when installing the asyncio reactor.Using custom asyncio loops==========================-You can also use custom asyncio event loops with the asyncio reactor. Set the-:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event-loop class to use it instead of the default asyncio event loop.+You can use custom asyncio event loops. Set the :setting:`ASYNCIO_EVENT_LOOP`+setting to the import path of the desired event loop class to use it instead of+the default asyncio event loop.+++.. _asyncio-old-versions:++Supporting older Scrapy versions+================================++Scrapy before 2.13.0 doesn't use the asyncio reactor by default, and users of+newer versions can opt out of it, so if you have code that uses asyncio APIs+you should decide which configurations to support.++.. _enforce-asyncio-requirement:++Enforcing asyncio as a requirement+----------------------------------++If your code requires asyncio to work, or you want to make it simpler, use+:func:`scrapy.utils.asyncio.is_asyncio_available` to :ref:`enforce asyncio as a+requirement <enforce-component-requirements>`. For example:++.. code-block:: python++ try:+ from scrapy.utils.asyncio import is_asyncio_available+ except ImportError: # Scrapy < 2.14+ from scrapy.utils.reactor import (+ is_asyncio_reactor_installed as is_asyncio_available,+ )+++ class MyComponent:+ def __init__(self):+ if not is_asyncio_available():+ raise ValueError(f"{MyComponent.__qualname__} requires asyncio support.")++With this in place you can safely write code that assumes asyncio support, e.g.+code that calls asyncio APIs, awaits on futures and uses+:func:`~scrapy.utils.defer.deferred_to_future` to await on Deferreds.++.. autofunction:: scrapy.utils.asyncio.is_asyncio_available++Making asyncio optional+-----------------------++If, instead, you also want to support configurations where asyncio isn't+available, you cannot use asyncio-specific APIs directly. This includes+:func:`~scrapy.utils.defer.deferred_to_future`; you should use the following+function instead:++.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future++You can also write two branches of the same code for asyncio and non-asyncio+environments, switching between them using+:func:`~scrapy.utils.asyncio.is_asyncio_available`.++Additionally, Scrapy provides unified helpers for some useful APIs that+automatically choose a Twisted-based implementation when asyncio is not+available:++.. autofunction:: scrapy.utils.asyncio.sleep+.. autofunction:: scrapy.utils.asyncio.call_later+.. autofunction:: scrapy.utils.asyncio.create_looping_call+.. autofunction:: scrapy.utils.asyncio.run_in_thread+Using newer helpers+--------------------.. _disable-asyncio:+Many of the asyncio-related Scrapy helpers are only available in some Scrapy+versions; in that case their documentation shows the first version where that+specific helper is available. If you want to use a helper with a Scrapy version+where it's not available, you can simply copy the relevant code into your+project.-Switching to a non-asyncio reactor-==================================+When you want to check whether the Scrapy version used provides a given helper,+prefer a ``try ... except ImportError`` approach to comparing the versions+directly, as future Scrapy versions will drop some of these helpers.-If for some reason your code doesn't work with the asyncio reactor, you can use-a different reactor by setting the :setting:`TWISTED_REACTOR` setting to its-import path (e.g. ``'twisted.internet.epollreactor.EPollReactor'``) or to-``None``, which will use the default reactor for your platform. If you are-using :class:`~scrapy.crawler.AsyncCrawlerRunner` or-:class:`~scrapy.crawler.AsyncCrawlerProcess` you also need to switch to their-Deferred-based counterparts: :class:`~scrapy.crawler.CrawlerRunner` or-:class:`~scrapy.crawler.CrawlerProcess` respectively.+You can also :ref:`require a specific minimum Scrapy version+<enforce-component-requirements>`.diff --git a/docs/topics/commands.rst b/docs/topics/commands.rstindex 2a8d6779b..ca221b936 100644--- a/docs/topics/commands.rst+++ b/docs/topics/commands.rst@@ -791,29 +791,8 @@ spider or a special internal one:* :command:`shell`* :command:`view`-They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess` or-:class:`scrapy.crawler.CrawlerProcess` for this. In most cases, this detail-shouldn't matter to the user running the command, but when the user :ref:`needs-a non-default Twisted reactor <disable-asyncio>`, it may be important.--Scrapy decides which of these two classes to use based on the value of the-:setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED` settings.-With :setting:`TWISTED_REACTOR_ENABLED` set to ``False`` it will use-:class:`~scrapy.crawler.AsyncCrawlerProcess`. Otherwise, if the-:setting:`TWISTED_REACTOR` value is the default one-(``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``),-:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used, otherwise-:class:`~scrapy.crawler.CrawlerProcess` will be used. The :ref:`spider settings-<spider-settings>` are not taken into account when doing this, as they are-loaded after this decision is made. This may cause an error if the-project-level setting is set to :ref:`the asyncio reactor <install-asyncio>`-(:ref:`explicitly <project-settings>` or :ref:`by using the Scrapy default-<default-settings>`) and :ref:`the setting of the spider being run-<spider-settings>` is set to :ref:`a different one <disable-asyncio>`, because-:class:`~scrapy.crawler.AsyncCrawlerProcess` only supports the asyncio reactor.-In this case, you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to-``True`` (at the project level or via the command line) so that Scrapy uses-:class:`~scrapy.crawler.CrawlerProcess`, which supports all reactors.+They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess`+for this... _custom-commands:@@ -841,9 +820,8 @@ and overriding specific methods. Here's what you need to know:* :attr:`~scrapy.commands.ScrapyCommand.requires_project` (bool): If ``True``,the command only runs inside a Scrapy project (default: ``False``).* :attr:`~scrapy.commands.ScrapyCommand.requires_crawler_process` (bool): If- ``True``, a :class:`~scrapy.crawler.AsyncCrawlerProcess` or- :class:`~scrapy.crawler.CrawlerProcess` instance will be created by Scrapy- when the command runs and made available in the+ ``True``, a :class:`~scrapy.crawler.AsyncCrawlerProcess` instance will be+ created by Scrapy when the command runs and made available in the:attr:`~scrapy.commands.ScrapyCommand.crawler_process` attribute (default:``True``).* :attr:`~scrapy.commands.ScrapyCommand.default_settings` (dict): Settings thatdiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 683fa7751..58974c22a 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -124,21 +124,17 @@ In the future we plan to add support for the ``async def`` syntax to these APIsor replace them with other APIs where changing the existing ones isn'tpossible.-These APIs have a coroutine-based implementation and a Deferred-based one:+These APIs have a coroutine-based implementation and a Deferred-based one, to+make it easier to write Deferred-based code; the Deferred-based ones will be+deprecated in a future Scrapy version:-- :class:`scrapy.crawler.Crawler`:-- - :meth:`~scrapy.crawler.Crawler.crawl_async` (coroutine-based) and- :meth:`~scrapy.crawler.Crawler.crawl` (Deferred-based): the former- may be inconvenient to use in Deferred-based code so both are available,- this may change in a future Scrapy version.+- :meth:`~scrapy.crawler.Crawler.crawl_async` (coroutine-based) and+ :meth:`~scrapy.crawler.Crawler.crawl` (Deferred-based)- :class:`scrapy.crawler.AsyncCrawlerRunner` and its subclass:class:`scrapy.crawler.AsyncCrawlerProcess` (coroutine-based) and:class:`scrapy.crawler.CrawlerRunner` and its subclass- :class:`scrapy.crawler.CrawlerProcess` (Deferred-based): the former- doesn't support non-default reactors and so the latter should be used- with those.+ :class:`scrapy.crawler.CrawlerProcess` (Deferred-based)The following user-supplied methods can return:class:`~twisted.internet.defer.Deferred` objects (the methods that can also@@ -203,12 +199,13 @@ becomes:.. code-block:: pythonfrom itemadapter import ItemAdapter+ from scrapy.utils.defer import deferred_to_futureclass DbPipeline:async def process_item(self, item):adapter = ItemAdapter(item)- adapter["field"] = await db.get_some_data(adapter["id"])+ adapter["field"] = await deferred_to_future(db.get_some_data(adapter["id"]))return itemCoroutines may be used to call asynchronous code. This includes other@@ -237,38 +234,16 @@ means you can use many useful Python libraries that provide such code:additional_data = await additional_response.text()# ... use response and additional_data to yield items and requests-.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the- :mod:`asyncio` loop, and to use them you need to :doc:`enable asyncio- support in Scrapy<asyncio>`.--.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,- you need to :ref:`wrap them<asyncio-await-dfd>`.+.. note:: If you want to ``await`` on Deferreds, you need to :ref:`wrap them+ <asyncio-await-dfd>`.Sometimes you need to call code that is not async-friendly at all, such as athird-party client library with no asynchronous support. Calling such codedirectly from a coroutine would block the underlying event loop, preventingScrapy from doing any other work while it runs. To avoid this, you can wrap-the blocking call with :func:`~scrapy.utils.decorators.inthread`, which runs-it in a separate thread and returns an awaitable with the result:--.. skip: next-.. code-block:: python-- from scrapy.utils.decorators import inthread--- @inthread- def blocking_call(value):- # e.g. a call into a library with no async support- return some_sync_client.get(value)--- class MySpider(Spider):- async def parse(self, response):- result = await deferred_to_future(blocking_call(response.url))- # ... use result to yield items and requests--.. autofunction:: scrapy.utils.decorators.inthread+the blocking call with :func:`asyncio.to_thread`, which runs it in a separate+thread and returns an awaitable with the result (see :ref:`optimize-blocking`+for an example).Common use cases for asynchronous code include:@@ -280,8 +255,6 @@ Common use cases for asynchronous code include:* delaying the spider initialization until some external event (in the:signal:`spider_opened` handler).-.. _aio-libs: https://github.com/aio-libs-.. _inline-requests:diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rstindex f1c0a019b..f891e74ec 100644--- a/docs/topics/optimize.rst+++ b/docs/topics/optimize.rst@@ -224,8 +224,9 @@ Move the slow code to a thread so that downloads continue while it runs:.. code-block:: python+ import asyncio+from scrapy import Spider- from scrapy.utils.asyncio import run_in_threadclass SlowParseSpider(Spider):@@ -233,7 +234,7 @@ Move the slow code to a thread so that downloads continue while it runs:start_urls = ["https://toscrape.com"]async def parse(self, response):- yield await run_in_thread(self.extract_item, response)+ yield await asyncio.to_thread(self.extract_item, response)def extract_item(self, response):# CPU-heavy parsing@@ -342,9 +343,6 @@ Other tips.. _uvloop: https://github.com/MagicStack/uvloop- Alternatively, try :ref:`switching to a non-asyncio reactor- <disable-asyncio>`.-- Disable unused :ref:`components <topics-components>`.For example, set :setting:`COOKIES_ENABLED` to ``False`` unless you needdiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex bd973dfab..4484166bd 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -317,7 +317,6 @@ These settings are:- :setting:`ADDONS`- :setting:`COMMANDS_MODULE`-- :setting:`FORCE_CRAWLER_PROCESS`- :setting:`SPIDER_LOADER_CLASS` and settings used by the correspondingspider loader class, e.g. :setting:`SPIDER_MODULES` and:setting:`SPIDER_LOADER_WARN_ONLY` for the default spider loader class.@@ -327,10 +326,6 @@ These settings are:``update_pre_crawler_settings()`` method of :ref:`add-ons <topics-addons>`enabled that way is not called.-:setting:`TWISTED_REACTOR` also acts as a pre-crawler setting when running a-:ref:`command that needs a CrawlerProcess <topics-commands-crawlerprocess>`,-since its project-level value determines the crawler process class.-.. _reactor-settings:Reactor settings@@ -349,22 +344,23 @@ Because only 1 reactor can be used per process, these settings cannot use adifferent value per spider when :ref:`running multiple spiders in the sameprocess <run-multiple-spiders>`.-These settings are used upon installing the reactor:--- :setting:`ASYNCIO_EVENT_LOOP` (not possible to set per-spider when using- :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)--- :setting:`TWISTED_REACTOR` (ignored when using- :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)--They can be :ref:`set from a spider <spider-settings>`, but only the values-from the first spider that runs are used, since that is when the reactor is-installed. If a later spider asks for a different reactor or a different event-loop, an exception is raised. With-:class:`~scrapy.crawler.CrawlerRunner` and+The :setting:`ASYNCIO_EVENT_LOOP` setting is used upon installing the reactor.+It can be :ref:`set from a spider <spider-settings>`, but only the value+from the first spider that runs is used, since that is when the reactor is+installed. If a later spider asks for a different event loop, an exception is+raised. With :class:`~scrapy.crawler.CrawlerRunner` and:class:`~scrapy.crawler.AsyncCrawlerRunner` the reactor must be installed-beforehand, so these settings are only used to check that the installed reactor-and event loop match them.+beforehand, so this setting is only used to check that the installed event loop+matches it.++There is an additional restriction when using+:class:`~scrapy.crawler.AsyncCrawlerProcess`: when this class is instantiated,+it installs :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`,+using the value of :setting:`ASYNCIO_EVENT_LOOP` that was passed to+:meth:`AsyncCrawlerProcess.__init__()+<scrapy.crawler.AsyncCrawlerProcess.__init__>`. If a different value for+:setting:`ASYNCIO_EVENT_LOOP` is provided later, e.g. in :ref:`per-spider+settings <spider-settings>`, an exception will be raised.These settings are applied when starting the reactor:@@ -380,18 +376,6 @@ different value, a warning is issued. They are ignored altogether when using:class:`~scrapy.crawler.CrawlerRunner` or:class:`~scrapy.crawler.AsyncCrawlerRunner`, which do not start the reactor.-There is an additional restriction for :setting:`TWISTED_REACTOR` and-:setting:`ASYNCIO_EVENT_LOOP` when using-:class:`~scrapy.crawler.AsyncCrawlerProcess`: when this class is instantiated,-it installs :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`,-ignoring the value of :setting:`TWISTED_REACTOR` and using the value of-:setting:`ASYNCIO_EVENT_LOOP` that was passed to-:meth:`AsyncCrawlerProcess.__init__()-<scrapy.crawler.AsyncCrawlerProcess.__init__>`. If a different value for-:setting:`TWISTED_REACTOR` or :setting:`ASYNCIO_EVENT_LOOP` is provided later,-e.g. in :ref:`per-spider settings <spider-settings>`, an exception will be-raised.-All of these settings, except for :setting:`ASYNCIO_EVENT_LOOP`, are only usedwhen the Twisted reactor is used, i.e. when :setting:`TWISTED_REACTOR_ENABLED`is ``True``.@@ -461,12 +445,9 @@ Default: ``None``Import path of a given ``asyncio`` event loop class.-If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) or when-:ref:`running Scrapy without a reactor <asyncio-without-reactor>` this setting-can be used to specify the-asyncio event loop to be used with it. Set the setting to the import path of the-desired asyncio event loop class. If the setting is set to ``None`` the default asyncio-event loop will be used.+This setting can be used to specify the asyncio event loop class to be used.+Set the setting to the import path of the desired asyncio event loop class. If+the setting is set to ``None`` the default asyncio event loop will be used.If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop@@ -1574,33 +1555,6 @@ Default: ``""``The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.-.. setting:: FORCE_CRAWLER_PROCESS--FORCE_CRAWLER_PROCESS------------------------Default: ``False``--If ``False``, :ref:`Scrapy commands that need a CrawlerProcess-<topics-commands-crawlerprocess>`, when :setting:`TWISTED_REACTOR_ENABLED`-is set to ``True``, will decide between using-:class:`scrapy.crawler.AsyncCrawlerProcess` and-:class:`scrapy.crawler.CrawlerProcess` based on the value of the-:setting:`TWISTED_REACTOR` setting, but ignoring its value in :ref:`per-spider-settings <spider-settings>`.--If ``True``, these commands will always use-:class:`~scrapy.crawler.CrawlerProcess` when :setting:`TWISTED_REACTOR_ENABLED`-is set to ``True``.--When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``,-:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used in all cases.--Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a-non-default value in :ref:`per-spider settings <spider-settings>`.--.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.-.. reqmeta:: ftp_passive.. setting:: FTP_PASSIVE_MODE@@ -2550,8 +2504,7 @@ Default: ``True``Whether to install and use the Twisted reactor.If this is set to ``True``, Scrapy will use the Twisted reactor and will-install one according to the :setting:`TWISTED_REACTOR` setting value when-appropriate (e.g. when running via :ref:`the command-line tool+install one when appropriate (e.g. when running via :ref:`the command-line tool<topics-commands>`). This is the traditional mode of using Scrapy.If this is set to ``False``, Scrapy will use the asyncio event loop directly@@ -2569,114 +2522,6 @@ See :ref:`asyncio-without-reactor` for more information about this mode... versionadded:: 2.15.0-.. setting:: TWISTED_REACTOR--TWISTED_REACTOR------------------Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``--Import path of a given :mod:`~twisted.internet.reactor`.--.. note::- This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.--Scrapy will install this reactor if no other reactor is installed yet, such as-when the ``scrapy`` CLI program is invoked or when using the-:class:`~scrapy.crawler.AsyncCrawlerProcess` class or the-:class:`~scrapy.crawler.CrawlerProcess` class.--If you are using the :class:`~scrapy.crawler.AsyncCrawlerRunner` class or the-:class:`~scrapy.crawler.CrawlerRunner` class, you also-need to install the correct reactor manually. You can do that using-:func:`~scrapy.utils.reactor.install_reactor`:--.. autofunction:: scrapy.utils.reactor.install_reactor--If a reactor is already installed,-:func:`~scrapy.utils.reactor.install_reactor` has no effect.--:class:`~scrapy.crawler.AsyncCrawlerRunner` and other similar classes raise an-exception if the installed reactor does not match the-:setting:`TWISTED_REACTOR` setting; therefore, having top-level-:mod:`~twisted.internet.reactor` imports in project files and imported-third-party libraries will make Scrapy raise an exception when it checks which-reactor is installed.--In order to use the reactor installed by Scrapy:--.. skip: next-.. code-block:: python-- import scrapy- from twisted.internet import reactor--- class QuotesSpider(scrapy.Spider):- name = "quotes"-- def __init__(self, *args, **kwargs):- self.timeout = int(kwargs.pop("timeout", "60"))- super().__init__(*args, **kwargs)-- async def start(self):- reactor.callLater(self.timeout, self.stop)-- urls = ["https://quotes.toscrape.com/page/1"]- for url in urls:- yield scrapy.Request(url=url, callback=self.parse)-- def parse(self, response):- for quote in response.css("div.quote"):- yield {"text": quote.css("span.text::text").get()}-- def stop(self):- self.crawler.engine.close_spider(self, "timeout")---which raises an exception, becomes:--.. code-block:: python-- import scrapy--- class QuotesSpider(scrapy.Spider):- name = "quotes"-- def __init__(self, *args, **kwargs):- self.timeout = int(kwargs.pop("timeout", "60"))- super().__init__(*args, **kwargs)-- async def start(self):- from twisted.internet import reactor-- reactor.callLater(self.timeout, self.stop)-- urls = ["https://quotes.toscrape.com/page/1"]- for url in urls:- yield scrapy.Request(url=url, callback=self.parse)-- def parse(self, response):- for quote in response.css("div.quote"):- yield {"text": quote.css("span.text::text").get()}-- def stop(self):- self.crawler.engine.close_spider(self, "timeout")---If this setting is set ``None``, Scrapy will use the existing reactor if one is-already installed, or install the default reactor defined by Twisted for the-current platform.--.. versionchanged:: 2.13- The default value was changed from ``None`` to- ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.--For additional information, see :doc:`core/howto/choosing-reactor`.--.. note:: This is a :ref:`reactor setting <reactor-settings>`.-.. setting:: UPLOAD_TIMEOUTUPLOAD_TIMEOUTdiff --git a/scrapy/cmdline.py b/scrapy/cmdline.pyindex 1f18e21c4..74eb71a37 100644--- a/scrapy/cmdline.py+++ b/scrapy/cmdline.py@@ -5,13 +5,14 @@ import cProfileimport inspectimport osimport sys+import warningsfrom importlib.metadata import entry_pointsfrom typing import TYPE_CHECKING, ParamSpecimport scrapyfrom scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatterfrom scrapy.crawler import AsyncCrawlerProcess, CrawlerProcess-from scrapy.exceptions import UsageError+from scrapy.exceptions import ScrapyDeprecationWarning, UsageErrorfrom scrapy.utils.misc import walk_modules_iterfrom scrapy.utils.project import get_project_settings, inside_projectfrom scrapy.utils.reactor import _asyncio_reactor_path@@ -211,9 +212,15 @@ def execute(argv: list[str] | None = None, settings: Settings | None = None) ->_run_print_help(parser, cmd.process_options, args, opts)if cmd.requires_crawler_process:+ if force_crawler_process := settings.getbool("FORCE_CRAWLER_PROCESS"):+ warnings.warn(+ "The FORCE_CRAWLER_PROCESS setting is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )if (settings["TWISTED_REACTOR"] == _asyncio_reactor_path- and not settings.getbool("FORCE_CRAWLER_PROCESS")+ and not force_crawler_process) or not settings.getbool("TWISTED_REACTOR_ENABLED"):cmd.crawler_process = AsyncCrawlerProcess(settings)else:diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.pyindex a556530e5..6aa0e9c9a 100644--- a/scrapy/core/downloader/handlers/_base_streaming.py+++ b/scrapy/core/downloader/handlers/_base_streaming.py@@ -78,13 +78,7 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, Generic[_ResponseT])def __init__(self, crawler: Crawler):if self.requires_asyncio and not is_asyncio_available(): # pragma: no cover- raise NotConfigured(- f"{type(self).__name__} requires the asyncio support. Make"- f" sure that you have either enabled the asyncio Twisted"- f" reactor in the TWISTED_REACTOR setting or disabled the"- f" TWISTED_REACTOR_ENABLED setting. See the asyncio documentation"- f" of Scrapy for more information."- )+ raise NotConfigured(f"{type(self).__name__} requires the asyncio support.")self._check_deps_installed()super().__init__(crawler)if self.experimental:diff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex d5501a96a..99cc524fb 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -33,8 +33,10 @@ from scrapy.utils.log import (log_scrapy_info,)from scrapy.utils.misc import build_from_crawler, load_object+from scrapy.utils.python import global_object_namefrom scrapy.utils.reactor import (_asyncio_reactor_path,+ _install_reactor,_is_asyncio_reactor_installed,install_reactor,is_reactor_installed,@@ -181,7 +183,7 @@ class Crawler:_configure_logger_levels(self.settings)install_scrapy_root_handler(self.settings)- def _apply_settings(self) -> None:+ def _apply_settings(self) -> None: # noqa: PLR0912if self.settings.frozen:return@@ -212,10 +214,10 @@ class Crawler:# but before something imports twisted.internet.reactor.if reactor_class:# Install a specific reactor.- install_reactor(reactor_class, event_loop)+ _install_reactor(reactor_class, event_loop)else:# Install the default one.- from twisted.internet import reactor # noqa: F401+ from twisted.internet import reactorelif not is_reactor_installed():# We need a reactor to be already installed.raise RuntimeError(@@ -229,6 +231,25 @@ class Crawler:if self._init_reactor or reactor_class:log_reactor_info()++ if not _is_asyncio_reactor_installed():+ from twisted.internet import reactor++ warnings.warn(+ f"Using a Twisted reactor other than {_asyncio_reactor_path}"+ f" is deprecated (the installed reactor is {global_object_name(type(reactor))})."+ " Remove any custom TWISTED_REACTOR setting and make sure that"+ " a different reactor is not installed by code outside Scrapy.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ elif not reactor_class:+ # TWISTED_REACTOR = None or "" (for "-s TWISTED_REACTOR="), asyncio reactor pre-installed+ warnings.warn(+ 'Setting TWISTED_REACTOR to None or "" is deprecated.',+ ScrapyDeprecationWarning,+ stacklevel=2,+ )else:# We expect a reactor to not be installed.if is_reactor_installed():diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.pyindex 2754ba209..29daafdba 100644--- a/scrapy/utils/asyncio.py+++ b/scrapy/utils/asyncio.py@@ -64,21 +64,10 @@ def is_asyncio_available() -> bool:``False`` if a different reactor is installed, and raises a:exc:`RuntimeError` if no reactor is installed.- Code that doesn't directly require a Twisted reactor should use this- function while code that requires- :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` should- also use :func:`~scrapy.utils.reactor.is_reactor_installed`.-- When this returns ``True``, an asyncio loop is installed and used by- Scrapy. It's possible to call functions that require it, such as- :func:`asyncio.sleep`, and await on :class:`asyncio.Future` objects in- Scrapy-related code.-- When this returns ``False``, a non-asyncio Twisted reactor is installed.- It's not possible to use asyncio features that require an asyncio event- loop or await on :class:`asyncio.Future` objects in Scrapy-related code,- but it's possible to await on :class:`~twisted.internet.defer.Deferred`- objects.+ The case when this function returns ``False`` is deprecated, and in a+ future Scrapy version this function will always return ``True``. Use it+ only if you need to support non-asyncio reactors or if you want to provide+ a better error message on those... note:: As this function uses :func:`asyncio.get_running_loop()`, it willonly detect the event loop if called in the same thread and from the@@ -171,13 +160,13 @@ class AsyncioLoopingCall(Generic[_P, _T]):some API and behavior compatibility with:class:`~twisted.internet.task.LoopingCall`.+ .. versionadded:: 2.14.0+The function is called every *interval* seconds, independent of the finishtime of the previous call. If the function is still running when it's timeto call it again, calls are skipped until the function finishes.The function must not return a coroutine or a ``Deferred``.-- .. versionadded:: 2.14.0"""def __init__(self, func: Callable[_P, _T], *args: _P.args, **kwargs: _P.kwargs):@@ -263,12 +252,15 @@ def create_looping_call() -> AsyncioLoopingCall[_P, _T] | LoopingCall:"""Create an instance of a looping call class.+ .. versionadded:: 2.14.0+This creates an instance of- :class:`~scrapy.utils.asyncio.AsyncioLoopingCall` or- :class:`~twisted.internet.task.LoopingCall`, depending on whether asyncio+ :class:`scrapy.utils.asyncio.AsyncioLoopingCall` or+ :class:`twisted.internet.task.LoopingCall`, depending on whether asynciosupport is available.- .. versionadded:: 2.14.0+ Use this function only if you need to support non-asyncio reactors. It will+ be deprecated in a later Scrapy version."""if is_asyncio_available():return AsyncioLoopingCall(func, *args, **kwargs)@@ -280,11 +272,14 @@ def call_later() -> CallLaterResult:"""Schedule a function to be called after a delay.+ .. versionadded:: 2.14.0+This uses either :meth:`asyncio.loop.call_later` or:meth:`reactor.callLater() <twisted.internet.base.ReactorBase.callLater>`,depending on whether asyncio support is available.- .. versionadded:: 2.14.0+ Use this function only if you need to support non-asyncio reactors. It will+ be deprecated in a later Scrapy version."""if is_asyncio_available():loop = _get_running_or_installed_loop()@@ -302,12 +297,12 @@ class CallLaterResult:"""An universal result for :func:`call_later`, wrapping either:class:`asyncio.TimerHandle` or :class:`twisted.internet.base.DelayedCall`.+ .. versionadded:: 2.14.0+The provided API is close to the :class:`asyncio.TimerHandle` one: there isno ``active()`` (as there is no such public API in:class:`asyncio.TimerHandle`) but ``cancel()`` can be called on alreadycalled or cancelled instances.-- .. versionadded:: 2.14.0"""_timer_handle: asyncio.TimerHandle | None = None@@ -346,8 +341,11 @@ async def sleep(seconds: float) -> None:.. versionadded:: 2.18.0This uses either :func:`asyncio.sleep` or- :func:`~twisted.internet.task.deferLater`, depending on whether asyncio+ :func:`twisted.internet.task.deferLater`, depending on whether asynciosupport is available.++ Use this function only if you need to support non-asyncio reactors. It will+ be deprecated in a later Scrapy version."""if is_asyncio_available():await asyncio.sleep(seconds)@@ -363,11 +361,14 @@ async def run_in_thread() -> _T:"""Call a function in a thread and return its result as a coroutine.+ .. versionadded:: 2.15.0+This uses either :func:`asyncio.to_thread` or:func:`twisted.internet.threads.deferToThread`, depending on whetherasyncio support is available.- .. versionadded:: 2.15.0+ Use this function only if you need to support non-asyncio reactors. It will+ be deprecated in a later Scrapy version."""if is_asyncio_available():return await asyncio.to_thread(func, *args, **kwargs)diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.pyindex 131dc1276..2bc4ad244 100644--- a/scrapy/utils/defer.py+++ b/scrapy/utils/defer.py@@ -630,10 +630,11 @@ def deferred_to_future(d: Deferred[_T]) -> Future[_T]:This function requires an installed asyncio reactor or a running asyncioevent loop, see :ref:`using-asyncio`.- In this state you cannot await on :class:`~twisted.internet.defer.Deferred`- objects from :ref:`Scrapy callables defined as coroutines- <coroutine-support>`, you can only await on ``Future`` objects. Wrapping- ``Deferred`` objects into ``Future`` objects allows you to wait on them:+ This function is needed because you cannot await on+ :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy+ callables defined as coroutines <coroutine-support>`; you can only await on+ :class:`~asyncio.Future` objects. Wrapping ``Deferred`` objects into+ ``Future`` objects allows you to await on them:.. code-block:: python@@ -661,27 +662,12 @@ def maybe_deferred_to_future(d: Deferred[_T]) -> Deferred[_T] | Future[_T]:"""Return *d* as an object that can be awaited from a :ref:`Scrapy callabledefined as a coroutine <coroutine-support>`.- What you can await in Scrapy callables defined as coroutines depends on the- value of :setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED`:+ This is similar to :func:`deferred_to_future`, but when a non-asyncio+ Twisted reactor is installed (which is deprecated), it returns *d* as is,+ because in that case it can be awaited directly.- - When :ref:`using the asyncio reactor <install-asyncio>`, or :ref:`not- using a reactor at all <asyncio-without-reactor>`, you can only await- on :class:`asyncio.Future` objects.-- - When :ref:`using a non-asyncio reactor <disable-asyncio>`, you can only- await on :class:`~twisted.internet.defer.Deferred` objects.-- If you want to write code that uses ``Deferred`` objects but works in both- of these states, use this function on all ``Deferred`` objects:-- .. code-block:: python-- class MySpider(Spider):- ...-- async def parse(self, response):- deferred = some_dfd_helper()- result = await maybe_deferred_to_future(deferred)+ Use this function only if you need to support non-asyncio reactors. It will+ be deprecated in a later Scrapy version."""if not is_asyncio_available():return ddiff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.pyindex b04130c43..d0f86de90 100644--- a/scrapy/utils/reactor.py+++ b/scrapy/utils/reactor.py@@ -124,16 +124,33 @@ def set_asyncio_event_loop_policy() -> None:def install_reactor(- reactor_path: str | None = None, event_loop_path: str | None = None+ reactor_path: str | None = None,+ event_loop_path: str | None = None,) -> None:- """Installs the :mod:`~twisted.internet.reactor` with the specified- import path. Also installs the asyncio event loop with the specified import- path if the asyncio reactor is enabled.+ """Install the asyncio Twisted reactor.- If `reactor_path` is `None`, the asyncio reactor will be installed.+ .. versionchanged:: VERSION+ *reactor_path* is now optional and accepts ``None``.++ *reactor_path* can be ``None`` or+ ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``; these values+ have the same effect.++ *event_loop_path*, if not ``None``, should be the import path of the+ asyncio event loop class to be used."""if reactor_path is None:reactor_path = _asyncio_reactor_path+ elif load_object(reactor_path) is not asyncioreactor.AsyncioSelectorReactor:+ warnings.warn(+ "Passing a non-asyncio reactor to install_reactor() is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ _install_reactor(reactor_path, event_loop_path)+++def _install_reactor(reactor_path: str, event_loop_path: str | None) -> None:reactor_class = load_object(reactor_path)if reactor_class is asyncioreactor.AsyncioSelectorReactor:set_asyncio_event_loop_policy()@@ -237,7 +254,8 @@ def is_reactor_installed() -> bool:def _is_asyncio_reactor_installed() -> bool:- """Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.+ """Check whether the installed reactor is+ :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.Raise a :exc:`RuntimeError` if no reactor is installed."""@@ -252,17 +270,18 @@ def _is_asyncio_reactor_installed() -> bool:def is_asyncio_reactor_installed() -> bool:- """Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.+ """Check whether the installed reactor is+ :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.Raise a :exc:`RuntimeError` if no reactor is installed."""warnings.warn("is_asyncio_reactor_installed() is deprecated."" If you need to check for asyncio support, use is_asyncio_available()."- " If you need to check whether a reactor is installed or not,"- " use is_reactor_installed() or is_reactorless()."- " If you need to check that specifically the asyncio reactor (or the non-asyncio one)"- " is installed, use is_reactor_installed() and is_asyncio_available().",+ " If you need to check whether a reactor is installed,"+ " use is_reactor_installed()."+ " See https://docs.scrapy.org/en/latest/topics/asyncio.html#asyncio-old-versions"+ " about supporting older Scrapy versions.",ScrapyDeprecationWarning,stacklevel=2,)diff --git a/scrapy/utils/test.py b/scrapy/utils/test.pyindex 8f11e70aa..12c4cc798 100644--- a/scrapy/utils/test.py+++ b/scrapy/utils/test.py@@ -36,7 +36,7 @@ def get_reactor_settings() -> dict[str, Any]:``Crawler._apply_settings()`` checks that the installed reactor matches thesettings, so tests that run the crawler in the current process may need to- pass a correct :setting:`TWISTED_REACTOR` setting value when creating it.+ pass a correct ``TWISTED_REACTOR`` setting value when creating it."""settings: dict[str, Any] = {}if is_reactor_installed():diff --git a/tests/CrawlerRunner/install_reactor_select.py b/tests/CrawlerRunner/install_reactor_select.pynew file mode 100644index 000000000..3073d9968--- /dev/null+++ b/tests/CrawlerRunner/install_reactor_select.py@@ -0,0 +1,30 @@+from twisted.internet.task import react++from scrapy import Spider+from scrapy.crawler import CrawlerRunner+from scrapy.utils.log import configure_logging+from scrapy.utils.reactor import install_reactor+++class NoRequestsSpider(Spider):+ name = "no_request"++ custom_settings = {+ "TWISTED_REACTOR": "twisted.internet.selectreactor.SelectReactor",+ }++ async def start(self):+ return+ yield+++def main(reactor):+ configure_logging(+ {"LOG_FORMAT": "%(levelname)s: %(message)s", "LOG_LEVEL": "DEBUG"}+ )+ runner = CrawlerRunner()+ return runner.crawl(NoRequestsSpider)+++install_reactor("twisted.internet.selectreactor.SelectReactor")+react(main)diff --git a/tests/test_command_crawl.py b/tests/test_command_crawl.pyindex 1ba6c9fbb..1e0aea5fe 100644--- a/tests/test_command_crawl.py+++ b/tests/test_command_crawl.py@@ -199,6 +199,10 @@ class MySpider(scrapy.Spider):"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"not in log)+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )assert "Spider closed (finished)" in logdef test_no_reactor(self, proj_path: Path) -> None:diff --git a/tests/test_command_runspider.py b/tests/test_command_runspider.pyindex 266b8dc59..bca79e321 100644--- a/tests/test_command_runspider.py+++ b/tests/test_command_runspider.py@@ -309,6 +309,10 @@ class MySpider(scrapy.Spider):"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"not in log)+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )@pytest.mark.requires_uvloopdef test_custom_asyncio_loop_enabled_true(self, tmp_path: Path) -> None:diff --git a/tests/test_commands.py b/tests/test_commands.pyindex b0b10b7bc..6de1ac668 100644--- a/tests/test_commands.py+++ b/tests/test_commands.py@@ -482,7 +482,11 @@ class MySpider(scrapy.Spider):"{'TWISTED_REACTOR': 'twisted.internet.selectreactor.SelectReactor'}",)- self._assert_spider_works(self.NORMAL_MSG, proj_path, "sp")+ _, _, err = proc(self.name, "sp", cwd=proj_path)+ assert self.NORMAL_MSG in err+ assert "It works!" in err+ assert "Spider closed (finished)" in err+ assert "FORCE_CRAWLER_PROCESS setting is deprecated" in errself._assert_spider_asyncio_fail(self.NORMAL_MSG, proj_path, "aiosp")diff --git a/tests/test_crawler.py b/tests/test_crawler.pyindex 430a98b76..6fd861799 100644--- a/tests/test_crawler.py+++ b/tests/test_crawler.py@@ -133,6 +133,28 @@ class TestCrawler:crawler._apply_settings()assert crawler.settings.getint(setting) == 3+ @pytest.mark.only_not_asyncio+ def test_non_asyncio_reactor_deprecated(self) -> None:+ with pytest.warns(+ ScrapyDeprecationWarning,+ match="Using a Twisted reactor other than "+ "twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated",+ ):+ get_crawler()++ @pytest.mark.parametrize("setting", [None, ""])+ @pytest.mark.only_asyncio+ @pytest.mark.requires_reactor+ def test_twisted_reactor_empty_deprecated(self, setting: str | None) -> None:+ with pytest.warns(+ ScrapyDeprecationWarning,+ match='Setting TWISTED_REACTOR to None or "" is deprecated',+ ) as record:+ get_crawler(settings_dict={"TWISTED_REACTOR": setting})+ assert not any(+ "Using a Twisted reactor other than" in str(w.message) for w in record+ )+def test_crawler_accepts_dict(self) -> None:crawler = get_crawler(DefaultSpider, {"foo": "bar"})assert crawler.settings["foo"] == "bar"diff --git a/tests/test_crawler_subprocess.py b/tests/test_crawler_subprocess.pyindex 1ff4de66a..80b2ca62a 100644--- a/tests/test_crawler_subprocess.py+++ b/tests/test_crawler_subprocess.py@@ -98,6 +98,10 @@ class TestCrawlerProcessSubprocessBase(ScriptRunnerMixin):in log)assert "RuntimeError" not in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ not in log+ )def test_ipv6_default_name_resolver(self) -> None:log = self.run_script("default_name_resolver.py")@@ -321,6 +325,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):# also starts using the select reactor by default in a future# version of Twisted), then we will need to rethink this test.assert "Spider closed (finished)" in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )else:assert "Spider closed (finished)" not in logassert (@@ -340,6 +348,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):log = self.run_script("reactor_select_twisted_reactor_select.py")assert "Spider closed (finished)" in logassert "ReactorAlreadyInstalledError" not in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )def test_reactor_select_subclass_twisted_reactor_select(self) -> None:log = self.run_script("reactor_select_subclass_twisted_reactor_select.py")@@ -353,6 +365,11 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):log = self.run_script("twisted_reactor_select.py")assert "Spider closed (finished)" in logassert "Using reactor: twisted.internet.selectreactor.SelectReactor" in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )+ assert "install_reactor() is deprecated" not in log@pytest.mark.skipif(platform.system() == "Windows", reason="PollReactor is not supported on Windows"@@ -361,6 +378,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):log = self.run_script("twisted_reactor_poll.py")assert "Spider closed (finished)" in logassert "Using reactor: twisted.internet.pollreactor.PollReactor" in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )def test_twisted_reactor_asyncio_custom_settings_conflict(self) -> None:log = self.run_script("twisted_reactor_custom_settings_conflict.py")@@ -369,6 +390,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):"(twisted.internet.selectreactor.SelectReactor) does not match the requested one"in log)+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )def test_reactorless(self) -> None:log = self.run_script("reactorless.py")@@ -625,6 +650,20 @@ class TestCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"not in log)+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )++ def test_install_reactor_select(self) -> None:+ log = self.run_script("install_reactor_select.py")+ assert "Spider closed (finished)" in log+ assert "Using reactor: twisted.internet.selectreactor.SelectReactor" in log+ assert "Passing a non-asyncio reactor to install_reactor() is deprecated" in log+ assert (+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"+ in log+ )def test_response_ip_address(self) -> None:log = self.run_script("ip_address.py")diff --git a/tests/test_feedexport_async.py b/tests/test_feedexport_async.pyindex bca51e523..d007ab5ae 100644--- a/tests/test_feedexport_async.py+++ b/tests/test_feedexport_async.py@@ -30,7 +30,7 @@ async def suspend() -> None:"""Let other coroutines run before resuming the calling one.Unlike :func:`asyncio.sleep`, this works with any value of the- :setting:`TWISTED_REACTOR` setting.+ ``TWISTED_REACTOR`` setting."""d: Deferred[None] = Deferred()call_later(0, d.callback, None)