Scrapy, a fast high-level web crawling & scraping framework for Python.
Document how DOWNLOAD_DELAY affects per-domain concurrency
Adrián Chaves34e4ed72ea87
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 2046c6446..468975d6a 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -633,25 +633,41 @@ DOWNLOAD_DELAYDefault: ``0``-The amount of time (in secs) that the downloader should wait before downloading-consecutive pages from the same website. This can be used to throttle the-crawling speed to avoid hitting servers too hard. Decimal numbers are-supported. Example::+Minimum seconds to wait between 2 consecutive requests to the same domain.- DOWNLOAD_DELAY = 0.25 # 250 ms of delay+Use :setting:`DOWNLOAD_DELAY` to throttle your crawling speed, to avoid hitting+servers too hard.++Decimal numbers are supported. For example, to send a maximum of 4 requests+every 10 seconds::++ DOWNLOAD_DELAY = 2.5This setting is also affected by the :setting:`RANDOMIZE_DOWNLOAD_DELAY`-setting (which is enabled by default). By default, Scrapy doesn't wait a fixed-amount of time between requests, but uses a random interval between 0.5 * :setting:`DOWNLOAD_DELAY` and 1.5 * :setting:`DOWNLOAD_DELAY`.+setting, which is enabled by default.When :setting:`CONCURRENT_REQUESTS_PER_IP` is non-zero, delays are enforced-per ip address instead of per domain.+per IP address instead of per domain.++Note that :setting:`DOWNLOAD_DELAY` can lower the effective per-domain+concurrency below :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`. If the response+time of a domain is lower than :setting:`DOWNLOAD_DELAY`, the effective+concurrency for that domain is 1. When testing throttling configurations, it+usually makes sense to lower :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` first,+and only increase :setting:`DOWNLOAD_DELAY` once+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` is 1 but a higher throttling is+desired... _spider-download_delay-attribute:-You can also change this setting per spider by setting ``download_delay``+You can change this setting per spider by setting the ``download_delay``spider attribute.+It is also possible to change this setting per domain, although it requires+non-trivial code. See the implementation of the :ref:`AutoThrottle+<topics-autothrottle>` extension for an example.++.. setting:: DOWNLOAD_HANDLERSDOWNLOAD_HANDLERS