Scrapy, a fast high-level web crawling & scraping framework for Python.
asyncio: cover accidental bad reactor installation, sort sections, reword the Windows section
Adrián Chaves72de48be6d35
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 7713b1af1..07baea071 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -27,54 +27,43 @@ reactor manually. You can do that usinginstall_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')-.. _using-custom-loops:+.. _asyncio-preinstalled-reactor:-Using custom asyncio loops-==========================+Handling a pre-installed reactor+================================-You can also use custom asyncio event loops with the asyncio reactor. Set the-:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to-use it instead of the default asyncio event loop.+``twisted.internet.reactor`` and some other Twisted imports install the default+Twisted reactor as a side effect. Once a Twisted reactor is installed, it is+not possible to switch to a different reactor at run time.+If you :ref:`configure the asyncio Twisted reactor <install-asyncio>` and, at+run time, Scrapy complains that a different reactor is already installed,+chances are you have some such imports in your code.-.. _asyncio-windows:+You can usually fix the issue by moving those offending module-level Twisted+imports to the method or function definitions where they are used. For example,+if you have something like:-Windows-specific notes-======================+.. code-block:: python-The Windows implementation of :mod:`asyncio` can use two event loop-implementations:+ from twisted.internet import reactor-- :class:`~asyncio.SelectorEventLoop`, default before Python 3.8, required- when using Twisted.-- :class:`~asyncio.ProactorEventLoop`, default since Python 3.8, cannot work- with Twisted.+ def my_function():+ reactor.callLater(...)-So on Python 3.8+ the event loop class needs to be changed.+Switch to something like:-.. versionchanged:: 2.6.0- The event loop class is changed automatically when you change the- :setting:`TWISTED_REACTOR` setting or call- :func:`~scrapy.utils.reactor.install_reactor`.+.. code-block:: python-To change the event loop class manually, call the following code before-installing the reactor::+ def my_function():+ from twisted.internet import reactor- import asyncio- asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())+ reactor.callLater(...)-You can put this in the same function that installs the reactor, if you do that-yourself, or in some code that runs before the reactor is installed, e.g.-``settings.py``.--.. note:: Other libraries you use may require- :class:`~asyncio.ProactorEventLoop`, e.g. because it supports- subprocesses (this is the case with `playwright`_), so you cannot use- them together with Scrapy on Windows (but you should be able to use- them on WSL or native Linux).--.. _playwright: https://github.com/microsoft/playwright-python+Alternatively, you can try to :ref:`manually install the asyncio reactor+<install-asyncio>`, with :func:`~scrapy.utils.reactor.install_reactor`, before+those imports happen... _asyncio-await-dfd:@@ -122,3 +111,36 @@ example:f"TWISTED_REACTOR setting. See the asyncio documentation "f"of Scrapy for more information.")+++.. _asyncio-windows:++Windows-specific notes+======================++The Windows implementation of :mod:`asyncio` can use two event loop+implementations, :class:`~asyncio.ProactorEventLoop` (default) and+:class:`~asyncio.SelectorEventLoop`. However, only+:class:`~asyncio.SelectorEventLoop` works with Twisted.++Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop`+automatically when you change the :setting:`TWISTED_REACTOR` setting or call+:func:`~scrapy.utils.reactor.install_reactor`.++.. note:: Other libraries you use may require+ :class:`~asyncio.ProactorEventLoop`, e.g. because it supports+ subprocesses (this is the case with `playwright`_), so you cannot use+ them together with Scrapy on Windows (but you should be able to use+ them on WSL or native Linux).++.. _playwright: https://github.com/microsoft/playwright-python+++.. _using-custom-loops:++Using custom asyncio loops+==========================++You can also use custom asyncio event loops with the asyncio reactor. Set the+:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event+loop class to use it instead of the default asyncio event loop.