Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Improve docs for multi-spider runs (#7907)

Adrian0b2d220197fb
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index dfa1e21f6..aeaf322c2 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -458,6 +458,18 @@ finishes before starting the next one:
should not have a different value per spider, and :ref:`pre-crawler
settings <pre-crawler-settings>` cannot be defined per spider.
+Every other setting applies to each crawler separately. This includes
+concurrency and politeness settings, such as :setting:`CONCURRENT_REQUESTS`,
+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`, and
+:ref:`AutoThrottle <topics-autothrottle>` also throttles each crawler
+separately. When crawling simultaneously, divide those values by the number of
+crawlers to keep the combined load on your hardware and on target websites
+unchanged.
+
+Because of this, running the same spider several times in the same process
+multiplies those limits instead of increasing crawling capacity. To crawl
+faster, raise :setting:`CONCURRENT_REQUESTS` on a single crawler.
+
.. seealso:: :ref:`run-from-script`.
.. skip: end
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 793d65f35..dc09fdeeb 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -754,6 +754,11 @@ Default: ``60``
Timeout for processing of DNS queries in seconds. Float is supported.
+The timeout starts when the query is queued into the Twisted reactor thread
+pool, not when it is sent. If that thread pool is saturated, queries can time
+out before being sent, in which case increasing
+:setting:`REACTOR_THREADPOOL_MAXSIZE` helps more than increasing this setting.
+
.. note::
This setting is only used by
:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when