Scrapy, a fast high-level web crawling & scraping framework for Python.
Improve docs for multi-spider runs (#7907)
Adrian0b2d220197fb
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex dfa1e21f6..aeaf322c2 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -458,6 +458,18 @@ finishes before starting the next one:should not have a different value per spider, and :ref:`pre-crawlersettings <pre-crawler-settings>` cannot be defined per spider.+Every other setting applies to each crawler separately. This includes+concurrency and politeness settings, such as :setting:`CONCURRENT_REQUESTS`,+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and :setting:`DOWNLOAD_DELAY`, and+:ref:`AutoThrottle <topics-autothrottle>` also throttles each crawler+separately. When crawling simultaneously, divide those values by the number of+crawlers to keep the combined load on your hardware and on target websites+unchanged.++Because of this, running the same spider several times in the same process+multiplies those limits instead of increasing crawling capacity. To crawl+faster, raise :setting:`CONCURRENT_REQUESTS` on a single crawler.+.. seealso:: :ref:`run-from-script`... skip: enddiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 793d65f35..dc09fdeeb 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -754,6 +754,11 @@ Default: ``60``Timeout for processing of DNS queries in seconds. Float is supported.+The timeout starts when the query is queued into the Twisted reactor thread+pool, not when it is sent. If that thread pool is saturated, queries can time+out before being sent, in which case increasing+:setting:`REACTOR_THREADPOOL_MAXSIZE` helps more than increasing this setting.+.. note::This setting is only used by:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when