Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Clarify the docs about reactor settings (#7880)

Adrianfc082aa914cf
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index de665f42e..de74ad5cf 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -69,9 +69,10 @@ Example::
precedence and override the project ones.
.. note:: :ref:`Pre-crawler settings <pre-crawler-settings>` cannot be defined
- per spider, and :ref:`reactor settings <reactor-settings>` should not have
- a different value per spider when :ref:`running multiple spiders in the
- same process <run-multiple-spiders>`.
+ per spider, and :ref:`reactor settings <reactor-settings>` and
+ :ref:`logging settings <logging-settings>` are subject to restrictions when
+ :ref:`running multiple spiders in the same process
+ <run-multiple-spiders>`.
One way to do so is by setting their :attr:`~scrapy.Spider.custom_settings`
attribute:
@@ -329,32 +330,41 @@ Reactor settings
**Reactor settings** are settings tied to the :doc:`Twisted reactor
<twisted:core/howto/reactor-basics>`.
-These settings can be defined from a spider. However, because only 1 reactor
-can be used per process, these settings cannot use a different value per spider
-when :ref:`running multiple spiders in the same process
-<run-multiple-spiders>`.
-
-In general, if different spiders define different values, the first defined
-value is used. However, if two spiders request a different reactor, an
-exception is raised.
+Because only 1 reactor can be used per process, these settings cannot use a
+different value per spider when :ref:`running multiple spiders in the same
+process <run-multiple-spiders>`.
-These settings are:
+These settings are used upon installing the reactor:
- :setting:`ASYNCIO_EVENT_LOOP` (not possible to set per-spider when using
:class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)
+- :setting:`TWISTED_REACTOR` (ignored when using
+ :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)
+
+They can be :ref:`set from a spider <spider-settings>`, but only the values
+from the first spider that runs are used, since that is when the reactor is
+installed. If a later spider asks for a different reactor or a different event
+loop, an exception is raised. With
+:class:`~scrapy.crawler.CrawlerRunner` and
+:class:`~scrapy.crawler.AsyncCrawlerRunner` the reactor must be installed
+beforehand, so these settings are only used to check that the installed reactor
+and event loop match them.
+
+These settings are applied when starting the reactor:
+
- :setting:`TWISTED_DNS_RESOLVER` and settings used by the corresponding
component, e.g. :setting:`DNSCACHE_ENABLED`, :setting:`DNSCACHE_SIZE`
and :setting:`DNS_TIMEOUT` for the default one.
- :setting:`REACTOR_THREADPOOL_MAXSIZE`
-- :setting:`TWISTED_REACTOR` (ignored when using
- :class:`~scrapy.crawler.AsyncCrawlerProcess`, see below)
-
-:setting:`ASYNCIO_EVENT_LOOP` and :setting:`TWISTED_REACTOR` are used upon
-installing the reactor. The rest of the settings are applied when starting
-the reactor.
+They are read from the settings of the
+:class:`~scrapy.crawler.CrawlerProcess` or
+:class:`~scrapy.crawler.AsyncCrawlerProcess` object, so setting them from a
+spider or an :ref:`add-on <topics-addons>` has no effect. They are ignored
+altogether when using :class:`~scrapy.crawler.CrawlerRunner` or
+:class:`~scrapy.crawler.AsyncCrawlerRunner`, which do not start the reactor.
There is an additional restriction for :setting:`TWISTED_REACTOR` and
:setting:`ASYNCIO_EVENT_LOOP` when using