Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document adjusting concurrency and delays from the telnet console (#8164)

Adrian14bf44840359
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst
index 3040bb283..1dbb5949f 100644
--- a/docs/topics/optimize.rst
+++ b/docs/topics/optimize.rst
@@ -158,7 +158,8 @@ crawl slower than a lower concurrency would have been. To find that limit:
- Crawl when the website is idle, in its own timezone, so that the capacity
you take is capacity nobody else wanted.
-- Raise concurrency gradually and watch the website respond.
+- Raise concurrency gradually and watch the website respond, changing it on
+ the running crawl from the :ref:`telnet console <telnet-concurrency>`.
:stat:`downloader/response_status_count/{status_code}` counts for 429, 503
or the ban page of the website, growing :stat:`retry/count`, or a
:ref:`download latency <download-latency>` that climbs as you push harder,
diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst
index 745f9a9aa..f23bdf37d 100644
--- a/docs/topics/telnetconsole.rst
+++ b/docs/topics/telnetconsole.rst
@@ -151,6 +151,33 @@ To stop::
>>> engine.stop()
Connection closed by foreign host.
+
+.. _telnet-concurrency:
+
+Change concurrency and delays
+-----------------------------
+
+The downloader checks its concurrency and delay attributes for every request
+it sends, so you can speed a crawl up or slow it down while it runs::
+
+ telnet localhost 6023
+ >>> downloader = engine.downloader
+ >>> downloader.total_concurrency = 32
+ >>> for slot in downloader.slots.values():
+ ... slot.concurrency = 4
+ ... slot.delay = 0.5
+ ...
+ >>>
+
+``total_concurrency`` is :setting:`CONCURRENT_REQUESTS`, and ``slots`` maps the
+name of every :setting:`download slot <DOWNLOAD_SLOTS>` in use to an object
+whose ``concurrency`` and ``delay`` come from
+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (or
+:setting:`CONCURRENT_REQUESTS_PER_IP`) and :setting:`DOWNLOAD_DELAY`. A slot
+created later, e.g. for a domain not visited yet, starts from those settings
+again, and :ref:`AutoThrottle <topics-autothrottle>` recalculates ``delay``
+after every response.
+
.. skip: end
Telnet Console signals