Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Refactor the docs

Adrián Chavesdc3ebb6cf76d
diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rst
index cfd6440f2..5bd72fa15 100644
--- a/docs/topics/autothrottle.rst
+++ b/docs/topics/autothrottle.rst
@@ -21,9 +21,14 @@ Design goals
How it works
============
-AutoThrottle extension adjusts download delays dynamically to make spider send
-:setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent requests on average
-to each remote website.
+Scrapy allows defining the concurrency and delay of different download slots,
+e.g. through the :setting:`DOWNLOAD_SLOTS` setting. By default requests are
+assigned to slots based on their URL domain, although it is possible to
+customize the download slot of any request.
+
+The AutoThrottle extension adjusts the delay of each download slot dynamically,
+to make your spider send :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` concurrent
+requests on average to each remote website.
It uses download latency to compute the delays. The main idea is the
following: if a server needs ``latency`` seconds to respond, a client
@@ -47,25 +52,6 @@ effect, but there are some important differences:
AutoThrottle doesn't have these issues.
-.. reqmeta:: autothrottle_dont_adjust_delay
-
-Disabling the throttling of a request
-=====================================
-
-To disable AutoThrottle for a specific request, set the
-``autothrottle_dont_adjust_delay`` request metadata key to ``True``:
-
-.. code-block:: python
-
- from scrapy import Request
-
- Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})
-
-Note, however, that AutoThrottle still determines the starting delay of every
-slot by setting the ``download_delay`` attribute on the running spider. You
-might want to set a custom value for the ``delay`` attribute of the slot, e.g.
-using :setting:`DOWNLOAD_SLOTS`.
-
Throttling algorithm
====================
@@ -99,6 +85,33 @@ callback, for example, and unable to attend downloads. However, these latencies
should still give a reasonable estimate of how busy Scrapy (and ultimately, the
server) is, and this extension builds on that premise.
+.. reqmeta:: autothrottle_dont_adjust_delay
+
+Prevent specific requests from triggering slot delay adjustments
+================================================================
+
+AutoThrottle adjusts the delay of download slots based on the latencies of
+responses that belong to that download slot. The only exceptions are non-200
+responses, which are only taken into account to increase that delay, but
+ignored if they would decrease that delay.
+
+You can also set the ``autothrottle_dont_adjust_delay`` request metadata key to
+``True`` in any request to prevent its response latency from impacting the
+delay of its download slot:
+
+.. code-block:: python
+
+ from scrapy import Request
+
+ Request("https://example.com", meta={"autothrottle_dont_adjust_delay": True})
+
+Note, however, that AutoThrottle still determines the starting delay of every
+download slot by setting the ``download_delay`` attribute on the running
+spider. If you want AutoThrottle not to impact a download slot at all, in
+addition to setting this meta key in all requests that use that download slot,
+you might want to set a custom value for the ``delay`` attribute of that
+download slot, e.g. using :setting:`DOWNLOAD_SLOTS`.
+
Settings
========