Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add docs about TWISTED_REACTOR and other per-process settings.

Andrey Rakhmatullin940cc0776ff8
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 732eba587..bd0dd8ce0 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -102,6 +102,17 @@ reactor after ``MySpider`` has finished running.
d.addBoth(lambda _: reactor.stop())
reactor.run() # the script will block here until the crawling is finished
+.. note::
+ .. versionchanged:: VERSION
+
+ The Twisted reactor is now installed when
+ :meth:`~scrapy.crawler.CrawlerProcess.crawl` is first called, not when a
+ :class:`scrapy.crawler.CrawlerProcess` object is created. Because of this,
+ :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` are now
+ honored in :attr:`~scrapy.Spider.custom_settings`. In older Scrapy versions
+ they are silently ignored when set there and you need to set these settings
+ in some other way.
+
.. seealso:: :doc:`twisted:core/howto/reactor-basics`
.. _run-multiple-spiders:
@@ -193,6 +204,25 @@ Same example but running the spiders sequentially by chaining the deferreds:
crawl()
reactor.run() # the script will block here until the last crawl call is finished
+Different spiders can set different values for the same setting, but when they
+run in the same process it may be impossible, by design or because of some
+limitations, to use these different values. What happens in practice is
+different for different settings:
+
+* :setting:`SPIDER_LOADER_CLASS` and the ones used by its value
+ (:setting:`SPIDER_MODULES`, :setting:`SPIDER_LOADER_WARN_ONLY` for the
+ default one) cannot be read from the per-spider settings. These are applied
+ when the :class:`~scrapy.crawler.CrawlerRunner` or
+ :class:`~scrapy.crawler.CrawlerProcess` object is created.
+* For :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` the first
+ available value is used, and if a spider requests a different reactor an
+ exception will be raised. These are applied when the reactor is installed.
+* For :setting:`REACTOR_THREADPOOL_MAXSIZE`, :setting:`DNS_RESOLVER` and the
+ ones used by the resolver (:setting:`DNSCACHE_ENABLED`,
+ :setting:`DNSCACHE_SIZE`, :setting:`DNS_TIMEOUT` for ones included in Scrapy)
+ the first available value is used. These are applied when the reactor is
+ started.
+
.. seealso:: :ref:`run-from-script`.
.. _distributed-crawls: