Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Deprecate non-asyncio reactors (#8454)

Andrey Rakhmatullin00436ad5cfe6
diff --git a/conftest.py b/conftest.py
index f7e18ee44..d113322fe 100644
--- a/conftest.py
+++ b/conftest.py
@@ -201,6 +201,12 @@ def pytest_configure(config):
if config.getoption("--reactor") == "none":
install_reactor_import_hook()
+ if config.getoption("--reactor") == "default":
+ config.addinivalue_line(
+ "filterwarnings",
+ "ignore:Using a Twisted reactor other than:scrapy.exceptions.ScrapyDeprecationWarning",
+ )
+
# Test modules made up mostly of tests that spawn a subprocess per test (see
# tests/utils/cmdline.py). Collected in file order, they form one contiguous
diff --git a/docs/faq.rst b/docs/faq.rst
index 7bc8e879e..77660ebd0 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -388,8 +388,7 @@ How to deal with ``<class 'ValueError'>: filedescriptor out of range in select()
This issue `has been reported`_ to appear when running broad crawls in macOS, where the default
Twisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time.
-If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch
-to a different one in the same way.
+If you are using this reactor explicitly, switch to the default one.
.. _faq-stop-response-download:
diff --git a/docs/news.rst b/docs/news.rst
index b19033f8b..9db7a6a19 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -2355,9 +2355,9 @@ Modified requirements
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- If you set the :setting:`TWISTED_REACTOR` setting to a :ref:`non-asyncio
- value <disable-asyncio>` at the :ref:`spider level <spider-settings>`, you
- may now need to set the :setting:`FORCE_CRAWLER_PROCESS` setting to
+- If you set the ``TWISTED_REACTOR`` setting to a non-asyncio
+ value at the :ref:`spider level <spider-settings>`, you
+ may now need to set the ``FORCE_CRAWLER_PROCESS`` setting to
``True`` when running Scrapy via :ref:`its command-line tool
<topics-commands-crawlerprocess>` to avoid a reactor mismatch exception.
(:gh:`6845`)
diff --git a/docs/news/2.x.rst b/docs/news/2.x.rst
index 6164b0cbc..21647f288 100644
--- a/docs/news/2.x.rst
+++ b/docs/news/2.x.rst
@@ -205,13 +205,13 @@ Modified requirements
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- The default value of the :setting:`TWISTED_REACTOR` setting was changed
+- The default value of the ``TWISTED_REACTOR`` setting was changed
from ``None`` to
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
was used in newly generated projects since Scrapy 2.7.0 but now existing
projects that don't explicitly set this setting will also use the asyncio
- reactor. You can :ref:`change this setting in your project
- <disable-asyncio>` to use a different reactor.
+ reactor. You can change this setting in your project to use a different
+ reactor.
(:gh:`6659`, :gh:`6713`)
- The iteration of start requests and items no longer stops once there are
@@ -2460,7 +2460,7 @@ Also fixes the following regressions introduced in 2.6.0:
- Installing a Twisted reactor before Scrapy does (e.g. importing
:mod:`twisted.internet.reactor` somewhere at the module level) no longer
prevents Scrapy from starting, as long as a different reactor is not
- specified in :setting:`TWISTED_REACTOR` (:gh:`5525`, :gh:`5528`)
+ specified in ``TWISTED_REACTOR`` (:gh:`5525`, :gh:`5528`)
- Fixed an exception that was being logged after the spider finished under
certain conditions (:gh:`5437`, :gh:`5440`)
@@ -2739,7 +2739,7 @@ New features
Bug fixes
~~~~~~~~~
-- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings
+- The ``TWISTED_REACTOR`` and :setting:`ASYNCIO_EVENT_LOOP` settings
are no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`.
(:gh:`4485`, :gh:`5352`)
@@ -3914,7 +3914,7 @@ Scrapy 2.0.1 (2020-03-18)
* Removed top-level :mod:`~twisted.internet.reactor` imports to prevent
errors about the wrong Twisted reactor being installed when setting a
- different Twisted reactor using :setting:`TWISTED_REACTOR` (:gh:`4401`,
+ different Twisted reactor using ``TWISTED_REACTOR`` (:gh:`4401`,
:gh:`4406`)
* Fixed tests (:gh:`4422`)
@@ -4049,7 +4049,7 @@ New features
existing soft limit that pauses request downloads when the total response
data being processed is too high (:gh:`1410`, :gh:`3551`)
-* A new :setting:`TWISTED_REACTOR` setting allows customizing the
+* A new ``TWISTED_REACTOR`` setting allows customizing the
:mod:`~twisted.internet.reactor` that Scrapy uses, allowing to
:doc:`enable asyncio support <topics/asyncio>` or deal with a
:ref:`common macOS issue <faq-specific-reactor>` (:gh:`2905`,
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index 643e274ec..3acc90c6e 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -4,10 +4,8 @@
asyncio
=======
-Scrapy supports :mod:`asyncio` natively. New projects created with
-:command:`startproject` have asyncio enabled by default, and you can use
-:mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine
-<coroutines>`.
+Scrapy supports :mod:`asyncio` natively. You can use :mod:`asyncio` and
+:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.
The rest of this page covers advanced topics. If you are starting a new project,
no additional setup is needed.
@@ -15,28 +13,23 @@ no additional setup is needed.
.. _install-asyncio:
-Configuring the asyncio reactor
-===============================
+Installing the asyncio reactor
+==============================
-New projects generated with :command:`startproject` have the asyncio
-reactor configured by default. No manual setup is needed.
-
-The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy
-uses. Its default value is
-``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``, which enables
-:mod:`asyncio` support.
-
-If you are using :class:`~scrapy.crawler.AsyncCrawlerRunner` or
-:class:`~scrapy.crawler.CrawlerRunner`, you also need to
-install the :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`
-reactor manually. You can do that using
-:func:`~scrapy.utils.reactor.install_reactor`:
+When Scrapy installs a reactor, it will install the correct one automatically.
+If you are managing the reactor yourself, when using
+:class:`~scrapy.crawler.AsyncCrawlerRunner` or
+:class:`~scrapy.crawler.CrawlerRunner`, you need to install the
+:class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` reactor
+manually. You can do that using :func:`~scrapy.utils.reactor.install_reactor`:
.. skip: next
.. code-block:: python
install_reactor()
+.. autofunction:: scrapy.utils.reactor.install_reactor
+
.. _asyncio-preinstalled-reactor:
@@ -47,8 +40,7 @@ Handling a pre-installed reactor
Twisted reactor as a side effect. Once a Twisted reactor is installed, it is
not possible to switch to a different reactor at run time.
-If you :ref:`configure the asyncio Twisted reactor <install-asyncio>` and, at
-run time, Scrapy complains that a different reactor is already installed,
+If at run time Scrapy complains that a different reactor is already installed,
chances are you have some such imports in your code.
You can usually fix the issue by moving those offending module-level Twisted
@@ -84,21 +76,9 @@ Integrating Deferred code and asyncio code
==========================================
Coroutine functions can await on Deferreds by wrapping them into
-:class:`asyncio.Future` objects. Scrapy provides two helpers for this:
+:class:`asyncio.Future` objects. Scrapy provides a helper for this:
.. autofunction:: scrapy.utils.defer.deferred_to_future
-.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future
-
-.. tip:: If you don't need to support reactors other than the default
- :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`, you
- can use :func:`~scrapy.utils.defer.deferred_to_future`, otherwise you
- should use :func:`~scrapy.utils.defer.maybe_deferred_to_future`.
-
-.. tip:: If you need to use these functions in code that aims to be compatible
- with lower versions of Scrapy that do not provide these functions,
- down to Scrapy 2.0 (earlier versions do not support
- :mod:`asyncio`), you can copy the implementation of these functions
- into your own code.
Coroutines and futures can be wrapped into Deferreds (for example, when a
Scrapy API requires passing a Deferred to it) using the following helpers:
@@ -110,33 +90,7 @@ The following function helps with a reverse wrapping:
.. autofunction:: scrapy.utils.defer.ensure_awaitable
-
-.. _enforce-asyncio-requirement:
-
-Enforcing asyncio as a requirement
-==================================
-
-If you are writing a :ref:`component <topics-components>` that requires asyncio
-to work, use :func:`scrapy.utils.asyncio.is_asyncio_available` to
-:ref:`enforce it as a requirement <enforce-component-requirements>`. For
-example:
-
-.. code-block:: python
-
- from scrapy.utils.asyncio import is_asyncio_available
-
-
- class MyComponent:
- def __init__(self):
- if not is_asyncio_available():
- raise ValueError(
- f"{MyComponent.__qualname__} requires the asyncio support. "
- f"Make sure you have configured the asyncio reactor in the "
- f"TWISTED_REACTOR setting. See the asyncio documentation "
- f"of Scrapy for more information."
- )
-
-.. autofunction:: scrapy.utils.asyncio.is_asyncio_available
+.. seealso:: :ref:`asyncio-old-versions`
.. _asyncio-without-reactor:
@@ -248,8 +202,9 @@ Here are some examples of APIs and patterns that need a replacement:
execute code in other threads. You can use :func:`asyncio.to_thread`,
:meth:`asyncio.loop.call_soon_threadsafe` and related APIs instead.
* Using :class:`twisted.internet.task.LoopingCall` for scheduling repeated
- tasks. As there is no direct replacement in the standard library, you may
- need to write your own one using :func:`asyncio.sleep` in a task.
+ tasks. As there is no direct replacement in the standard library, you can use
+ :class:`scrapy.utils.asyncio.AsyncioLoopingCall` or write your own simple
+ implementation using :func:`asyncio.sleep` in a task.
* Using Twisted network client and server APIs (:meth:`reactor.connectTCP()
<twisted.internet.interfaces.IReactorTCP.connectTCP>`,
:meth:`reactor.listenTCP()
@@ -261,13 +216,10 @@ Here are some examples of APIs and patterns that need a replacement:
:class:`~scrapy.crawler.AsyncCrawlerProcess` or
:class:`~scrapy.crawler.AsyncCrawlerRunner` respectively instead.
-Scrapy provides unified helpers for some of these examples:
+Scrapy provides this replacement for
+:class:`~twisted.internet.task.LoopingCall`:
-.. autofunction:: scrapy.utils.asyncio.sleep
-.. autofunction:: scrapy.utils.asyncio.call_later
-.. autofunction:: scrapy.utils.asyncio.create_looping_call
.. autoclass:: scrapy.utils.asyncio.AsyncioLoopingCall
-.. autofunction:: scrapy.utils.asyncio.run_in_thread
If your code needs to know whether the reactor is available, you can either
check for the value of the :setting:`TWISTED_REACTOR_ENABLED` setting (you need
@@ -341,21 +293,85 @@ automatically when installing the asyncio reactor.
Using custom asyncio loops
==========================
-You can also use custom asyncio event loops with the asyncio reactor. Set the
-:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event
-loop class to use it instead of the default asyncio event loop.
+You can use custom asyncio event loops. Set the :setting:`ASYNCIO_EVENT_LOOP`
+setting to the import path of the desired event loop class to use it instead of
+the default asyncio event loop.
+
+
+.. _asyncio-old-versions:
+
+Supporting older Scrapy versions
+================================
+
+Scrapy before 2.13.0 doesn't use the asyncio reactor by default, and users of
+newer versions can opt out of it, so if you have code that uses asyncio APIs
+you should decide which configurations to support.
+
+.. _enforce-asyncio-requirement:
+
+Enforcing asyncio as a requirement
+----------------------------------
+
+If your code requires asyncio to work, or you want to make it simpler, use
+:func:`scrapy.utils.asyncio.is_asyncio_available` to :ref:`enforce asyncio as a
+requirement <enforce-component-requirements>`. For example:
+
+.. code-block:: python
+
+ try:
+ from scrapy.utils.asyncio import is_asyncio_available
+ except ImportError: # Scrapy < 2.14
+ from scrapy.utils.reactor import (
+ is_asyncio_reactor_installed as is_asyncio_available,
+ )
+
+
+ class MyComponent:
+ def __init__(self):
+ if not is_asyncio_available():
+ raise ValueError(f"{MyComponent.__qualname__} requires asyncio support.")
+
+With this in place you can safely write code that assumes asyncio support, e.g.
+code that calls asyncio APIs, awaits on futures and uses
+:func:`~scrapy.utils.defer.deferred_to_future` to await on Deferreds.
+
+.. autofunction:: scrapy.utils.asyncio.is_asyncio_available
+
+Making asyncio optional
+-----------------------
+
+If, instead, you also want to support configurations where asyncio isn't
+available, you cannot use asyncio-specific APIs directly. This includes
+:func:`~scrapy.utils.defer.deferred_to_future`; you should use the following
+function instead:
+
+.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future
+
+You can also write two branches of the same code for asyncio and non-asyncio
+environments, switching between them using
+:func:`~scrapy.utils.asyncio.is_asyncio_available`.
+
+Additionally, Scrapy provides unified helpers for some useful APIs that
+automatically choose a Twisted-based implementation when asyncio is not
+available:
+
+.. autofunction:: scrapy.utils.asyncio.sleep
+.. autofunction:: scrapy.utils.asyncio.call_later
+.. autofunction:: scrapy.utils.asyncio.create_looping_call
+.. autofunction:: scrapy.utils.asyncio.run_in_thread
+Using newer helpers
+-------------------
-.. _disable-asyncio:
+Many of the asyncio-related Scrapy helpers are only available in some Scrapy
+versions; in that case their documentation shows the first version where that
+specific helper is available. If you want to use a helper with a Scrapy version
+where it's not available, you can simply copy the relevant code into your
+project.
-Switching to a non-asyncio reactor
-==================================
+When you want to check whether the Scrapy version used provides a given helper,
+prefer a ``try ... except ImportError`` approach to comparing the versions
+directly, as future Scrapy versions will drop some of these helpers.
-If for some reason your code doesn't work with the asyncio reactor, you can use
-a different reactor by setting the :setting:`TWISTED_REACTOR` setting to its
-import path (e.g. ``'twisted.internet.epollreactor.EPollReactor'``) or to
-``None``, which will use the default reactor for your platform. If you are
-using :class:`~scrapy.crawler.AsyncCrawlerRunner` or
-:class:`~scrapy.crawler.AsyncCrawlerProcess` you also need to switch to their
-Deferred-based counterparts: :class:`~scrapy.crawler.CrawlerRunner` or
-:class:`~scrapy.crawler.CrawlerProcess` respectively.
+You can also :ref:`require a specific minimum Scrapy version
+<enforce-component-requirements>`.
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index 2a8d6779b..ca221b936 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -791,29 +791,8 @@ spider or a special internal one:
* :command:`shell`
* :command:`view`
-They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess` or
-:class:`scrapy.crawler.CrawlerProcess` for this. In most cases, this detail
-shouldn't matter to the user running the command, but when the user :ref:`needs
-a non-default Twisted reactor <disable-asyncio>`, it may be important.
-
-Scrapy decides which of these two classes to use based on the value of the
-:setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED` settings.
-With :setting:`TWISTED_REACTOR_ENABLED` set to ``False`` it will use
-:class:`~scrapy.crawler.AsyncCrawlerProcess`. Otherwise, if the
-:setting:`TWISTED_REACTOR` value is the default one
-(``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``),
-:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used, otherwise
-:class:`~scrapy.crawler.CrawlerProcess` will be used. The :ref:`spider settings
-<spider-settings>` are not taken into account when doing this, as they are
-loaded after this decision is made. This may cause an error if the
-project-level setting is set to :ref:`the asyncio reactor <install-asyncio>`
-(:ref:`explicitly <project-settings>` or :ref:`by using the Scrapy default
-<default-settings>`) and :ref:`the setting of the spider being run
-<spider-settings>` is set to :ref:`a different one <disable-asyncio>`, because
-:class:`~scrapy.crawler.AsyncCrawlerProcess` only supports the asyncio reactor.
-In this case, you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to
-``True`` (at the project level or via the command line) so that Scrapy uses
-:class:`~scrapy.crawler.CrawlerProcess`, which supports all reactors.
+They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess`
+for this.
.. _custom-commands:
@@ -841,9 +820,8 @@ and overriding specific methods. Here's what you need to know:
* :attr:`~scrapy.commands.ScrapyCommand.requires_project` (bool): If ``True``,
the command only runs inside a Scrapy project (default: ``False``).
* :attr:`~scrapy.commands.ScrapyCommand.requires_crawler_process` (bool): If
- ``True``, a :class:`~scrapy.crawler.AsyncCrawlerProcess` or
- :class:`~scrapy.crawler.CrawlerProcess` instance will be created by Scrapy
- when the command runs and made available in the
+ ``True``, a :class:`~scrapy.crawler.AsyncCrawlerProcess` instance will be
+ created by Scrapy when the command runs and made available in the
:attr:`~scrapy.commands.ScrapyCommand.crawler_process` attribute (default:
``True``).
* :attr:`~scrapy.commands.ScrapyCommand.default_settings` (dict): Settings that
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 683fa7751..58974c22a 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -124,21 +124,17 @@ In the future we plan to add support for the ``async def`` syntax to these APIs
or replace them with other APIs where changing the existing ones isn't
possible.
-These APIs have a coroutine-based implementation and a Deferred-based one:
+These APIs have a coroutine-based implementation and a Deferred-based one, to
+make it easier to write Deferred-based code; the Deferred-based ones will be
+deprecated in a future Scrapy version:
-- :class:`scrapy.crawler.Crawler`:
-
- - :meth:`~scrapy.crawler.Crawler.crawl_async` (coroutine-based) and
- :meth:`~scrapy.crawler.Crawler.crawl` (Deferred-based): the former
- may be inconvenient to use in Deferred-based code so both are available,
- this may change in a future Scrapy version.
+- :meth:`~scrapy.crawler.Crawler.crawl_async` (coroutine-based) and
+ :meth:`~scrapy.crawler.Crawler.crawl` (Deferred-based)
- :class:`scrapy.crawler.AsyncCrawlerRunner` and its subclass
:class:`scrapy.crawler.AsyncCrawlerProcess` (coroutine-based) and
:class:`scrapy.crawler.CrawlerRunner` and its subclass
- :class:`scrapy.crawler.CrawlerProcess` (Deferred-based): the former
- doesn't support non-default reactors and so the latter should be used
- with those.
+ :class:`scrapy.crawler.CrawlerProcess` (Deferred-based)
The following user-supplied methods can return
:class:`~twisted.internet.defer.Deferred` objects (the methods that can also
@@ -203,12 +199,13 @@ becomes:
.. code-block:: python
from itemadapter import ItemAdapter
+ from scrapy.utils.defer import deferred_to_future
class DbPipeline:
async def process_item(self, item):
adapter = ItemAdapter(item)
- adapter["field"] = await db.get_some_data(adapter["id"])
+ adapter["field"] = await deferred_to_future(db.get_some_data(adapter["id"]))
return item
Coroutines may be used to call asynchronous code. This includes other
@@ -237,38 +234,16 @@ means you can use many useful Python libraries that provide such code:
additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests
-.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
- :mod:`asyncio` loop, and to use them you need to :doc:`enable asyncio
- support in Scrapy<asyncio>`.
-
-.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,
- you need to :ref:`wrap them<asyncio-await-dfd>`.
+.. note:: If you want to ``await`` on Deferreds, you need to :ref:`wrap them
+ <asyncio-await-dfd>`.
Sometimes you need to call code that is not async-friendly at all, such as a
third-party client library with no asynchronous support. Calling such code
directly from a coroutine would block the underlying event loop, preventing
Scrapy from doing any other work while it runs. To avoid this, you can wrap
-the blocking call with :func:`~scrapy.utils.decorators.inthread`, which runs
-it in a separate thread and returns an awaitable with the result:
-
-.. skip: next
-.. code-block:: python
-
- from scrapy.utils.decorators import inthread
-
-
- @inthread
- def blocking_call(value):
- # e.g. a call into a library with no async support
- return some_sync_client.get(value)
-
-
- class MySpider(Spider):
- async def parse(self, response):
- result = await deferred_to_future(blocking_call(response.url))
- # ... use result to yield items and requests
-
-.. autofunction:: scrapy.utils.decorators.inthread
+the blocking call with :func:`asyncio.to_thread`, which runs it in a separate
+thread and returns an awaitable with the result (see :ref:`optimize-blocking`
+for an example).
Common use cases for asynchronous code include:
@@ -280,8 +255,6 @@ Common use cases for asynchronous code include:
* delaying the spider initialization until some external event (in the
:signal:`spider_opened` handler).
-.. _aio-libs: https://github.com/aio-libs
-
.. _inline-requests:
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst
index f1c0a019b..f891e74ec 100644
--- a/docs/topics/optimize.rst
+++ b/docs/topics/optimize.rst
@@ -224,8 +224,9 @@ Move the slow code to a thread so that downloads continue while it runs:
.. code-block:: python
+ import asyncio
+
from scrapy import Spider
- from scrapy.utils.asyncio import run_in_thread
class SlowParseSpider(Spider):
@@ -233,7 +234,7 @@ Move the slow code to a thread so that downloads continue while it runs:
start_urls = ["https://toscrape.com"]
async def parse(self, response):
- yield await run_in_thread(self.extract_item, response)
+ yield await asyncio.to_thread(self.extract_item, response)
def extract_item(self, response):
# CPU-heavy parsing
@@ -342,9 +343,6 @@ Other tips
.. _uvloop: https://github.com/MagicStack/uvloop
- Alternatively, try :ref:`switching to a non-asyncio reactor
- <disable-asyncio>`.
-
- Disable unused :ref:`components <topics-components>`.
For example, set :setting:`COOKIES_ENABLED` to ``False`` unless you need
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index bd973dfab..4484166bd 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -317,7 +317,6 @@ These settings are:
- :setting:`ADDONS`
- :setting:`COMMANDS_MODULE`
-- :setting:`FORCE_CRAWLER_PROCESS`
- :setting:`SPIDER_LOADER_CLASS` and settings used by the corresponding
spider loader class, e.g. :setting:`SPIDER_MODULES` and
:setting:`SPIDER_LOADER_WARN_ONLY` for the default spider loader class.
@@ -327,10 +326,6 @@ These settings are:
``update_pre_crawler_settings()`` method of :ref:`add-ons <topics-addons>`
enabled that way is not called.
-:setting:`TWISTED_REACTOR` also acts as a pre-crawler setting when running a
-:ref:`command that needs a CrawlerProcess <topics-commands-crawlerprocess>`,
-since its project-level value determines the crawler process class.
-
.. _reactor-settings:
Reactor settings
@@ -349,22 +344,23 @@ Because only 1 reactor can be used per process, these settings cannot use a
different value per spider when :ref:`running multiple spiders in the same
process <run-multiple-spiders>`.
-These settings are used upon installing the reactor:
-
-- :setting:`ASYNCIO_EVENT_LOOP` (not possible to set per-spider when using
- :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)
-
-- :setting:`TWISTED_REACTOR` (ignored when using
- :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)
-
-They can be :ref:`set from a spider <spider-settings>`, but only the values
-from the first spider that runs are used, since that is when the reactor is
-installed. If a later spider asks for a different reactor or a different event
-loop, an exception is raised. With
-:class:`~scrapy.crawler.CrawlerRunner` and
+The :setting:`ASYNCIO_EVENT_LOOP` setting is used upon installing the reactor.
+It can be :ref:`set from a spider <spider-settings>`, but only the value
+from the first spider that runs is used, since that is when the reactor is
+installed. If a later spider asks for a different event loop, an exception is
+raised. With :class:`~scrapy.crawler.CrawlerRunner` and
:class:`~scrapy.crawler.AsyncCrawlerRunner` the reactor must be installed
-beforehand, so these settings are only used to check that the installed reactor
-and event loop match them.
+beforehand, so this setting is only used to check that the installed event loop
+matches it.
+
+There is an additional restriction when using
+:class:`~scrapy.crawler.AsyncCrawlerProcess`: when this class is instantiated,
+it installs :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`,
+using the value of :setting:`ASYNCIO_EVENT_LOOP` that was passed to
+:meth:`AsyncCrawlerProcess.__init__()
+<scrapy.crawler.AsyncCrawlerProcess.__init__>`. If a different value for
+:setting:`ASYNCIO_EVENT_LOOP` is provided later, e.g. in :ref:`per-spider
+settings <spider-settings>`, an exception will be raised.
These settings are applied when starting the reactor:
@@ -380,18 +376,6 @@ different value, a warning is issued. They are ignored altogether when using
:class:`~scrapy.crawler.CrawlerRunner` or
:class:`~scrapy.crawler.AsyncCrawlerRunner`, which do not start the reactor.
-There is an additional restriction for :setting:`TWISTED_REACTOR` and
-:setting:`ASYNCIO_EVENT_LOOP` when using
-:class:`~scrapy.crawler.AsyncCrawlerProcess`: when this class is instantiated,
-it installs :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`,
-ignoring the value of :setting:`TWISTED_REACTOR` and using the value of
-:setting:`ASYNCIO_EVENT_LOOP` that was passed to
-:meth:`AsyncCrawlerProcess.__init__()
-<scrapy.crawler.AsyncCrawlerProcess.__init__>`. If a different value for
-:setting:`TWISTED_REACTOR` or :setting:`ASYNCIO_EVENT_LOOP` is provided later,
-e.g. in :ref:`per-spider settings <spider-settings>`, an exception will be
-raised.
-
All of these settings, except for :setting:`ASYNCIO_EVENT_LOOP`, are only used
when the Twisted reactor is used, i.e. when :setting:`TWISTED_REACTOR_ENABLED`
is ``True``.
@@ -461,12 +445,9 @@ Default: ``None``
Import path of a given ``asyncio`` event loop class.
-If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) or when
-:ref:`running Scrapy without a reactor <asyncio-without-reactor>` this setting
-can be used to specify the
-asyncio event loop to be used with it. Set the setting to the import path of the
-desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
-event loop will be used.
+This setting can be used to specify the asyncio event loop class to be used.
+Set the setting to the import path of the desired asyncio event loop class. If
+the setting is set to ``None`` the default asyncio event loop will be used.
If you are installing the asyncio reactor manually using the :func:`~scrapy.utils.reactor.install_reactor`
function, you can use the ``event_loop_path`` parameter to indicate the import path of the event loop
@@ -1574,33 +1555,6 @@ Default: ``""``
The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.
-.. setting:: FORCE_CRAWLER_PROCESS
-
-FORCE_CRAWLER_PROCESS
----------------------
-
-Default: ``False``
-
-If ``False``, :ref:`Scrapy commands that need a CrawlerProcess
-<topics-commands-crawlerprocess>`, when :setting:`TWISTED_REACTOR_ENABLED`
-is set to ``True``, will decide between using
-:class:`scrapy.crawler.AsyncCrawlerProcess` and
-:class:`scrapy.crawler.CrawlerProcess` based on the value of the
-:setting:`TWISTED_REACTOR` setting, but ignoring its value in :ref:`per-spider
-settings <spider-settings>`.
-
-If ``True``, these commands will always use
-:class:`~scrapy.crawler.CrawlerProcess` when :setting:`TWISTED_REACTOR_ENABLED`
-is set to ``True``.
-
-When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``,
-:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used in all cases.
-
-Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a
-non-default value in :ref:`per-spider settings <spider-settings>`.
-
-.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
-
.. reqmeta:: ftp_passive
.. setting:: FTP_PASSIVE_MODE
@@ -2550,8 +2504,7 @@ Default: ``True``
Whether to install and use the Twisted reactor.
If this is set to ``True``, Scrapy will use the Twisted reactor and will
-install one according to the :setting:`TWISTED_REACTOR` setting value when
-appropriate (e.g. when running via :ref:`the command-line tool
+install one when appropriate (e.g. when running via :ref:`the command-line tool
<topics-commands>`). This is the traditional mode of using Scrapy.
If this is set to ``False``, Scrapy will use the asyncio event loop directly
@@ -2569,114 +2522,6 @@ See :ref:`asyncio-without-reactor` for more information about this mode.
.. versionadded:: 2.15.0
-.. setting:: TWISTED_REACTOR
-
-TWISTED_REACTOR
----------------
-
-Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``
-
-Import path of a given :mod:`~twisted.internet.reactor`.
-
-.. note::
- This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
-
-Scrapy will install this reactor if no other reactor is installed yet, such as
-when the ``scrapy`` CLI program is invoked or when using the
-:class:`~scrapy.crawler.AsyncCrawlerProcess` class or the
-:class:`~scrapy.crawler.CrawlerProcess` class.
-
-If you are using the :class:`~scrapy.crawler.AsyncCrawlerRunner` class or the
-:class:`~scrapy.crawler.CrawlerRunner` class, you also
-need to install the correct reactor manually. You can do that using
-:func:`~scrapy.utils.reactor.install_reactor`:
-
-.. autofunction:: scrapy.utils.reactor.install_reactor
-
-If a reactor is already installed,
-:func:`~scrapy.utils.reactor.install_reactor` has no effect.
-
-:class:`~scrapy.crawler.AsyncCrawlerRunner` and other similar classes raise an
-exception if the installed reactor does not match the
-:setting:`TWISTED_REACTOR` setting; therefore, having top-level
-:mod:`~twisted.internet.reactor` imports in project files and imported
-third-party libraries will make Scrapy raise an exception when it checks which
-reactor is installed.
-
-In order to use the reactor installed by Scrapy:
-
-.. skip: next
-.. code-block:: python
-
- import scrapy
- from twisted.internet import reactor
-
-
- class QuotesSpider(scrapy.Spider):
- name = "quotes"
-
- def __init__(self, *args, **kwargs):
- self.timeout = int(kwargs.pop("timeout", "60"))
- super().__init__(*args, **kwargs)
-
- async def start(self):
- reactor.callLater(self.timeout, self.stop)
-
- urls = ["https://quotes.toscrape.com/page/1"]
- for url in urls:
- yield scrapy.Request(url=url, callback=self.parse)
-
- def parse(self, response):
- for quote in response.css("div.quote"):
- yield {"text": quote.css("span.text::text").get()}
-
- def stop(self):
- self.crawler.engine.close_spider(self, "timeout")
-
-
-which raises an exception, becomes:
-
-.. code-block:: python
-
- import scrapy
-
-
- class QuotesSpider(scrapy.Spider):
- name = "quotes"
-
- def __init__(self, *args, **kwargs):
- self.timeout = int(kwargs.pop("timeout", "60"))
- super().__init__(*args, **kwargs)
-
- async def start(self):
- from twisted.internet import reactor
-
- reactor.callLater(self.timeout, self.stop)
-
- urls = ["https://quotes.toscrape.com/page/1"]
- for url in urls:
- yield scrapy.Request(url=url, callback=self.parse)
-
- def parse(self, response):
- for quote in response.css("div.quote"):
- yield {"text": quote.css("span.text::text").get()}
-
- def stop(self):
- self.crawler.engine.close_spider(self, "timeout")
-
-
-If this setting is set ``None``, Scrapy will use the existing reactor if one is
-already installed, or install the default reactor defined by Twisted for the
-current platform.
-
-.. versionchanged:: 2.13
- The default value was changed from ``None`` to
- ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.
-
-For additional information, see :doc:`core/howto/choosing-reactor`.
-
-.. note:: This is a :ref:`reactor setting <reactor-settings>`.
-
.. setting:: UPLOAD_TIMEOUT
UPLOAD_TIMEOUT
diff --git a/scrapy/cmdline.py b/scrapy/cmdline.py
index 1f18e21c4..74eb71a37 100644
--- a/scrapy/cmdline.py
+++ b/scrapy/cmdline.py
@@ -5,13 +5,14 @@ import cProfile
import inspect
import os
import sys
+import warnings
from importlib.metadata import entry_points
from typing import TYPE_CHECKING, ParamSpec
import scrapy
from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
from scrapy.crawler import AsyncCrawlerProcess, CrawlerProcess
-from scrapy.exceptions import UsageError
+from scrapy.exceptions import ScrapyDeprecationWarning, UsageError
from scrapy.utils.misc import walk_modules_iter
from scrapy.utils.project import get_project_settings, inside_project
from scrapy.utils.reactor import _asyncio_reactor_path
@@ -211,9 +212,15 @@ def execute(argv: list[str] | None = None, settings: Settings | None = None) ->
_run_print_help(parser, cmd.process_options, args, opts)
if cmd.requires_crawler_process:
+ if force_crawler_process := settings.getbool("FORCE_CRAWLER_PROCESS"):
+ warnings.warn(
+ "The FORCE_CRAWLER_PROCESS setting is deprecated.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
if (
settings["TWISTED_REACTOR"] == _asyncio_reactor_path
- and not settings.getbool("FORCE_CRAWLER_PROCESS")
+ and not force_crawler_process
) or not settings.getbool("TWISTED_REACTOR_ENABLED"):
cmd.crawler_process = AsyncCrawlerProcess(settings)
else:
diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.py
index a556530e5..6aa0e9c9a 100644
--- a/scrapy/core/downloader/handlers/_base_streaming.py
+++ b/scrapy/core/downloader/handlers/_base_streaming.py
@@ -78,13 +78,7 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, Generic[_ResponseT])
def __init__(self, crawler: Crawler):
if self.requires_asyncio and not is_asyncio_available(): # pragma: no cover
- raise NotConfigured(
- f"{type(self).__name__} requires the asyncio support. Make"
- f" sure that you have either enabled the asyncio Twisted"
- f" reactor in the TWISTED_REACTOR setting or disabled the"
- f" TWISTED_REACTOR_ENABLED setting. See the asyncio documentation"
- f" of Scrapy for more information."
- )
+ raise NotConfigured(f"{type(self).__name__} requires the asyncio support.")
self._check_deps_installed()
super().__init__(crawler)
if self.experimental:
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index d5501a96a..99cc524fb 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -33,8 +33,10 @@ from scrapy.utils.log import (
log_scrapy_info,
)
from scrapy.utils.misc import build_from_crawler, load_object
+from scrapy.utils.python import global_object_name
from scrapy.utils.reactor import (
_asyncio_reactor_path,
+ _install_reactor,
_is_asyncio_reactor_installed,
install_reactor,
is_reactor_installed,
@@ -181,7 +183,7 @@ class Crawler:
_configure_logger_levels(self.settings)
install_scrapy_root_handler(self.settings)
- def _apply_settings(self) -> None:
+ def _apply_settings(self) -> None: # noqa: PLR0912
if self.settings.frozen:
return
@@ -212,10 +214,10 @@ class Crawler:
# but before something imports twisted.internet.reactor.
if reactor_class:
# Install a specific reactor.
- install_reactor(reactor_class, event_loop)
+ _install_reactor(reactor_class, event_loop)
else:
# Install the default one.
- from twisted.internet import reactor # noqa: F401
+ from twisted.internet import reactor
elif not is_reactor_installed():
# We need a reactor to be already installed.
raise RuntimeError(
@@ -229,6 +231,25 @@ class Crawler:
if self._init_reactor or reactor_class:
log_reactor_info()
+
+ if not _is_asyncio_reactor_installed():
+ from twisted.internet import reactor
+
+ warnings.warn(
+ f"Using a Twisted reactor other than {_asyncio_reactor_path}"
+ f" is deprecated (the installed reactor is {global_object_name(type(reactor))})."
+ " Remove any custom TWISTED_REACTOR setting and make sure that"
+ " a different reactor is not installed by code outside Scrapy.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ elif not reactor_class:
+ # TWISTED_REACTOR = None or "" (for "-s TWISTED_REACTOR="), asyncio reactor pre-installed
+ warnings.warn(
+ 'Setting TWISTED_REACTOR to None or "" is deprecated.',
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
else:
# We expect a reactor to not be installed.
if is_reactor_installed():
diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.py
index 2754ba209..29daafdba 100644
--- a/scrapy/utils/asyncio.py
+++ b/scrapy/utils/asyncio.py
@@ -64,21 +64,10 @@ def is_asyncio_available() -> bool:
``False`` if a different reactor is installed, and raises a
:exc:`RuntimeError` if no reactor is installed.
- Code that doesn't directly require a Twisted reactor should use this
- function while code that requires
- :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` should
- also use :func:`~scrapy.utils.reactor.is_reactor_installed`.
-
- When this returns ``True``, an asyncio loop is installed and used by
- Scrapy. It's possible to call functions that require it, such as
- :func:`asyncio.sleep`, and await on :class:`asyncio.Future` objects in
- Scrapy-related code.
-
- When this returns ``False``, a non-asyncio Twisted reactor is installed.
- It's not possible to use asyncio features that require an asyncio event
- loop or await on :class:`asyncio.Future` objects in Scrapy-related code,
- but it's possible to await on :class:`~twisted.internet.defer.Deferred`
- objects.
+ The case when this function returns ``False`` is deprecated, and in a
+ future Scrapy version this function will always return ``True``. Use it
+ only if you need to support non-asyncio reactors or if you want to provide
+ a better error message on those.
.. note:: As this function uses :func:`asyncio.get_running_loop()`, it will
only detect the event loop if called in the same thread and from the
@@ -171,13 +160,13 @@ class AsyncioLoopingCall(Generic[_P, _T]):
some API and behavior compatibility with
:class:`~twisted.internet.task.LoopingCall`.
+ .. versionadded:: 2.14.0
+
The function is called every *interval* seconds, independent of the finish
time of the previous call. If the function is still running when it's time
to call it again, calls are skipped until the function finishes.
The function must not return a coroutine or a ``Deferred``.
-
- .. versionadded:: 2.14.0
"""
def __init__(self, func: Callable[_P, _T], *args: _P.args, **kwargs: _P.kwargs):
@@ -263,12 +252,15 @@ def create_looping_call(
) -> AsyncioLoopingCall[_P, _T] | LoopingCall:
"""Create an instance of a looping call class.
+ .. versionadded:: 2.14.0
+
This creates an instance of
- :class:`~scrapy.utils.asyncio.AsyncioLoopingCall` or
- :class:`~twisted.internet.task.LoopingCall`, depending on whether asyncio
+ :class:`scrapy.utils.asyncio.AsyncioLoopingCall` or
+ :class:`twisted.internet.task.LoopingCall`, depending on whether asyncio
support is available.
- .. versionadded:: 2.14.0
+ Use this function only if you need to support non-asyncio reactors. It will
+ be deprecated in a later Scrapy version.
"""
if is_asyncio_available():
return AsyncioLoopingCall(func, *args, **kwargs)
@@ -280,11 +272,14 @@ def call_later(
) -> CallLaterResult:
"""Schedule a function to be called after a delay.
+ .. versionadded:: 2.14.0
+
This uses either :meth:`asyncio.loop.call_later` or
:meth:`reactor.callLater() <twisted.internet.base.ReactorBase.callLater>`,
depending on whether asyncio support is available.
- .. versionadded:: 2.14.0
+ Use this function only if you need to support non-asyncio reactors. It will
+ be deprecated in a later Scrapy version.
"""
if is_asyncio_available():
loop = _get_running_or_installed_loop()
@@ -302,12 +297,12 @@ class CallLaterResult:
"""An universal result for :func:`call_later`, wrapping either
:class:`asyncio.TimerHandle` or :class:`twisted.internet.base.DelayedCall`.
+ .. versionadded:: 2.14.0
+
The provided API is close to the :class:`asyncio.TimerHandle` one: there is
no ``active()`` (as there is no such public API in
:class:`asyncio.TimerHandle`) but ``cancel()`` can be called on already
called or cancelled instances.
-
- .. versionadded:: 2.14.0
"""
_timer_handle: asyncio.TimerHandle | None = None
@@ -346,8 +341,11 @@ async def sleep(seconds: float) -> None:
.. versionadded:: 2.18.0
This uses either :func:`asyncio.sleep` or
- :func:`~twisted.internet.task.deferLater`, depending on whether asyncio
+ :func:`twisted.internet.task.deferLater`, depending on whether asyncio
support is available.
+
+ Use this function only if you need to support non-asyncio reactors. It will
+ be deprecated in a later Scrapy version.
"""
if is_asyncio_available():
await asyncio.sleep(seconds)
@@ -363,11 +361,14 @@ async def run_in_thread(
) -> _T:
"""Call a function in a thread and return its result as a coroutine.
+ .. versionadded:: 2.15.0
+
This uses either :func:`asyncio.to_thread` or
:func:`twisted.internet.threads.deferToThread`, depending on whether
asyncio support is available.
- .. versionadded:: 2.15.0
+ Use this function only if you need to support non-asyncio reactors. It will
+ be deprecated in a later Scrapy version.
"""
if is_asyncio_available():
return await asyncio.to_thread(func, *args, **kwargs)
diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py
index 131dc1276..2bc4ad244 100644
--- a/scrapy/utils/defer.py
+++ b/scrapy/utils/defer.py
@@ -630,10 +630,11 @@ def deferred_to_future(d: Deferred[_T]) -> Future[_T]:
This function requires an installed asyncio reactor or a running asyncio
event loop, see :ref:`using-asyncio`.
- In this state you cannot await on :class:`~twisted.internet.defer.Deferred`
- objects from :ref:`Scrapy callables defined as coroutines
- <coroutine-support>`, you can only await on ``Future`` objects. Wrapping
- ``Deferred`` objects into ``Future`` objects allows you to wait on them:
+ This function is needed because you cannot await on
+ :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy
+ callables defined as coroutines <coroutine-support>`; you can only await on
+ :class:`~asyncio.Future` objects. Wrapping ``Deferred`` objects into
+ ``Future`` objects allows you to await on them:
.. code-block:: python
@@ -661,27 +662,12 @@ def maybe_deferred_to_future(d: Deferred[_T]) -> Deferred[_T] | Future[_T]:
"""Return *d* as an object that can be awaited from a :ref:`Scrapy callable
defined as a coroutine <coroutine-support>`.
- What you can await in Scrapy callables defined as coroutines depends on the
- value of :setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED`:
+ This is similar to :func:`deferred_to_future`, but when a non-asyncio
+ Twisted reactor is installed (which is deprecated), it returns *d* as is,
+ because in that case it can be awaited directly.
- - When :ref:`using the asyncio reactor <install-asyncio>`, or :ref:`not
- using a reactor at all <asyncio-without-reactor>`, you can only await
- on :class:`asyncio.Future` objects.
-
- - When :ref:`using a non-asyncio reactor <disable-asyncio>`, you can only
- await on :class:`~twisted.internet.defer.Deferred` objects.
-
- If you want to write code that uses ``Deferred`` objects but works in both
- of these states, use this function on all ``Deferred`` objects:
-
- .. code-block:: python
-
- class MySpider(Spider):
- ...
-
- async def parse(self, response):
- deferred = some_dfd_helper()
- result = await maybe_deferred_to_future(deferred)
+ Use this function only if you need to support non-asyncio reactors. It will
+ be deprecated in a later Scrapy version.
"""
if not is_asyncio_available():
return d
diff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.py
index b04130c43..d0f86de90 100644
--- a/scrapy/utils/reactor.py
+++ b/scrapy/utils/reactor.py
@@ -124,16 +124,33 @@ def set_asyncio_event_loop_policy() -> None:
def install_reactor(
- reactor_path: str | None = None, event_loop_path: str | None = None
+ reactor_path: str | None = None,
+ event_loop_path: str | None = None,
) -> None:
- """Installs the :mod:`~twisted.internet.reactor` with the specified
- import path. Also installs the asyncio event loop with the specified import
- path if the asyncio reactor is enabled.
+ """Install the asyncio Twisted reactor.
- If `reactor_path` is `None`, the asyncio reactor will be installed.
+ .. versionchanged:: VERSION
+ *reactor_path* is now optional and accepts ``None``.
+
+ *reactor_path* can be ``None`` or
+ ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``; these values
+ have the same effect.
+
+ *event_loop_path*, if not ``None``, should be the import path of the
+ asyncio event loop class to be used.
"""
if reactor_path is None:
reactor_path = _asyncio_reactor_path
+ elif load_object(reactor_path) is not asyncioreactor.AsyncioSelectorReactor:
+ warnings.warn(
+ "Passing a non-asyncio reactor to install_reactor() is deprecated.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ _install_reactor(reactor_path, event_loop_path)
+
+
+def _install_reactor(reactor_path: str, event_loop_path: str | None) -> None:
reactor_class = load_object(reactor_path)
if reactor_class is asyncioreactor.AsyncioSelectorReactor:
set_asyncio_event_loop_policy()
@@ -237,7 +254,8 @@ def is_reactor_installed() -> bool:
def _is_asyncio_reactor_installed() -> bool:
- """Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
+ """Check whether the installed reactor is
+ :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
Raise a :exc:`RuntimeError` if no reactor is installed.
"""
@@ -252,17 +270,18 @@ def _is_asyncio_reactor_installed() -> bool:
def is_asyncio_reactor_installed() -> bool:
- """Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
+ """Check whether the installed reactor is
+ :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
Raise a :exc:`RuntimeError` if no reactor is installed.
"""
warnings.warn(
"is_asyncio_reactor_installed() is deprecated."
" If you need to check for asyncio support, use is_asyncio_available()."
- " If you need to check whether a reactor is installed or not,"
- " use is_reactor_installed() or is_reactorless()."
- " If you need to check that specifically the asyncio reactor (or the non-asyncio one)"
- " is installed, use is_reactor_installed() and is_asyncio_available().",
+ " If you need to check whether a reactor is installed,"
+ " use is_reactor_installed()."
+ " See https://docs.scrapy.org/en/latest/topics/asyncio.html#asyncio-old-versions"
+ " about supporting older Scrapy versions.",
ScrapyDeprecationWarning,
stacklevel=2,
)
diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py
index 8f11e70aa..12c4cc798 100644
--- a/scrapy/utils/test.py
+++ b/scrapy/utils/test.py
@@ -36,7 +36,7 @@ def get_reactor_settings() -> dict[str, Any]:
``Crawler._apply_settings()`` checks that the installed reactor matches the
settings, so tests that run the crawler in the current process may need to
- pass a correct :setting:`TWISTED_REACTOR` setting value when creating it.
+ pass a correct ``TWISTED_REACTOR`` setting value when creating it.
"""
settings: dict[str, Any] = {}
if is_reactor_installed():
diff --git a/tests/CrawlerRunner/install_reactor_select.py b/tests/CrawlerRunner/install_reactor_select.py
new file mode 100644
index 000000000..3073d9968
--- /dev/null
+++ b/tests/CrawlerRunner/install_reactor_select.py
@@ -0,0 +1,30 @@
+from twisted.internet.task import react
+
+from scrapy import Spider
+from scrapy.crawler import CrawlerRunner
+from scrapy.utils.log import configure_logging
+from scrapy.utils.reactor import install_reactor
+
+
+class NoRequestsSpider(Spider):
+ name = "no_request"
+
+ custom_settings = {
+ "TWISTED_REACTOR": "twisted.internet.selectreactor.SelectReactor",
+ }
+
+ async def start(self):
+ return
+ yield
+
+
+def main(reactor):
+ configure_logging(
+ {"LOG_FORMAT": "%(levelname)s: %(message)s", "LOG_LEVEL": "DEBUG"}
+ )
+ runner = CrawlerRunner()
+ return runner.crawl(NoRequestsSpider)
+
+
+install_reactor("twisted.internet.selectreactor.SelectReactor")
+react(main)
diff --git a/tests/test_command_crawl.py b/tests/test_command_crawl.py
index 1ba6c9fbb..1e0aea5fe 100644
--- a/tests/test_command_crawl.py
+++ b/tests/test_command_crawl.py
@@ -199,6 +199,10 @@ class MySpider(scrapy.Spider):
"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"
not in log
)
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
assert "Spider closed (finished)" in log
def test_no_reactor(self, proj_path: Path) -> None:
diff --git a/tests/test_command_runspider.py b/tests/test_command_runspider.py
index 266b8dc59..bca79e321 100644
--- a/tests/test_command_runspider.py
+++ b/tests/test_command_runspider.py
@@ -309,6 +309,10 @@ class MySpider(scrapy.Spider):
"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"
not in log
)
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
@pytest.mark.requires_uvloop
def test_custom_asyncio_loop_enabled_true(self, tmp_path: Path) -> None:
diff --git a/tests/test_commands.py b/tests/test_commands.py
index b0b10b7bc..6de1ac668 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -482,7 +482,11 @@ class MySpider(scrapy.Spider):
"{'TWISTED_REACTOR': 'twisted.internet.selectreactor.SelectReactor'}",
)
- self._assert_spider_works(self.NORMAL_MSG, proj_path, "sp")
+ _, _, err = proc(self.name, "sp", cwd=proj_path)
+ assert self.NORMAL_MSG in err
+ assert "It works!" in err
+ assert "Spider closed (finished)" in err
+ assert "FORCE_CRAWLER_PROCESS setting is deprecated" in err
self._assert_spider_asyncio_fail(self.NORMAL_MSG, proj_path, "aiosp")
diff --git a/tests/test_crawler.py b/tests/test_crawler.py
index 430a98b76..6fd861799 100644
--- a/tests/test_crawler.py
+++ b/tests/test_crawler.py
@@ -133,6 +133,28 @@ class TestCrawler:
crawler._apply_settings()
assert crawler.settings.getint(setting) == 3
+ @pytest.mark.only_not_asyncio
+ def test_non_asyncio_reactor_deprecated(self) -> None:
+ with pytest.warns(
+ ScrapyDeprecationWarning,
+ match="Using a Twisted reactor other than "
+ "twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated",
+ ):
+ get_crawler()
+
+ @pytest.mark.parametrize("setting", [None, ""])
+ @pytest.mark.only_asyncio
+ @pytest.mark.requires_reactor
+ def test_twisted_reactor_empty_deprecated(self, setting: str | None) -> None:
+ with pytest.warns(
+ ScrapyDeprecationWarning,
+ match='Setting TWISTED_REACTOR to None or "" is deprecated',
+ ) as record:
+ get_crawler(settings_dict={"TWISTED_REACTOR": setting})
+ assert not any(
+ "Using a Twisted reactor other than" in str(w.message) for w in record
+ )
+
def test_crawler_accepts_dict(self) -> None:
crawler = get_crawler(DefaultSpider, {"foo": "bar"})
assert crawler.settings["foo"] == "bar"
diff --git a/tests/test_crawler_subprocess.py b/tests/test_crawler_subprocess.py
index 1ff4de66a..80b2ca62a 100644
--- a/tests/test_crawler_subprocess.py
+++ b/tests/test_crawler_subprocess.py
@@ -98,6 +98,10 @@ class TestCrawlerProcessSubprocessBase(ScriptRunnerMixin):
in log
)
assert "RuntimeError" not in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ not in log
+ )
def test_ipv6_default_name_resolver(self) -> None:
log = self.run_script("default_name_resolver.py")
@@ -321,6 +325,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
# also starts using the select reactor by default in a future
# version of Twisted), then we will need to rethink this test.
assert "Spider closed (finished)" in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
else:
assert "Spider closed (finished)" not in log
assert (
@@ -340,6 +348,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
log = self.run_script("reactor_select_twisted_reactor_select.py")
assert "Spider closed (finished)" in log
assert "ReactorAlreadyInstalledError" not in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
def test_reactor_select_subclass_twisted_reactor_select(self) -> None:
log = self.run_script("reactor_select_subclass_twisted_reactor_select.py")
@@ -353,6 +365,11 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
log = self.run_script("twisted_reactor_select.py")
assert "Spider closed (finished)" in log
assert "Using reactor: twisted.internet.selectreactor.SelectReactor" in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
+ assert "install_reactor() is deprecated" not in log
@pytest.mark.skipif(
platform.system() == "Windows", reason="PollReactor is not supported on Windows"
@@ -361,6 +378,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
log = self.run_script("twisted_reactor_poll.py")
assert "Spider closed (finished)" in log
assert "Using reactor: twisted.internet.pollreactor.PollReactor" in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
def test_twisted_reactor_asyncio_custom_settings_conflict(self) -> None:
log = self.run_script("twisted_reactor_custom_settings_conflict.py")
@@ -369,6 +390,10 @@ class TestCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
"(twisted.internet.selectreactor.SelectReactor) does not match the requested one"
in log
)
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
def test_reactorless(self) -> None:
log = self.run_script("reactorless.py")
@@ -625,6 +650,20 @@ class TestCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):
"Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor"
not in log
)
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
+
+ def test_install_reactor_select(self) -> None:
+ log = self.run_script("install_reactor_select.py")
+ assert "Spider closed (finished)" in log
+ assert "Using reactor: twisted.internet.selectreactor.SelectReactor" in log
+ assert "Passing a non-asyncio reactor to install_reactor() is deprecated" in log
+ assert (
+ "Using a Twisted reactor other than twisted.internet.asyncioreactor.AsyncioSelectorReactor is deprecated"
+ in log
+ )
def test_response_ip_address(self) -> None:
log = self.run_script("ip_address.py")
diff --git a/tests/test_feedexport_async.py b/tests/test_feedexport_async.py
index bca51e523..d007ab5ae 100644
--- a/tests/test_feedexport_async.py
+++ b/tests/test_feedexport_async.py
@@ -30,7 +30,7 @@ async def suspend() -> None:
"""Let other coroutines run before resuming the calling one.
Unlike :func:`asyncio.sleep`, this works with any value of the
- :setting:`TWISTED_REACTOR` setting.
+ ``TWISTED_REACTOR`` setting.
"""
d: Deferred[None] = Deferred()
call_later(0, d.callback, None)