Scrapy, a fast high-level web crawling & scraping framework for Python.
Document adjusting concurrency and delays from the telnet console (#8164)
Adrian14bf44840359
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rstindex 3040bb283..1dbb5949f 100644--- a/docs/topics/optimize.rst+++ b/docs/topics/optimize.rst@@ -158,7 +158,8 @@ crawl slower than a lower concurrency would have been. To find that limit:- Crawl when the website is idle, in its own timezone, so that the capacityyou take is capacity nobody else wanted.-- Raise concurrency gradually and watch the website respond.+- Raise concurrency gradually and watch the website respond, changing it on+ the running crawl from the :ref:`telnet console <telnet-concurrency>`.:stat:`downloader/response_status_count/{status_code}` counts for 429, 503or the ban page of the website, growing :stat:`retry/count`, or a:ref:`download latency <download-latency>` that climbs as you push harder,diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rstindex 745f9a9aa..f23bdf37d 100644--- a/docs/topics/telnetconsole.rst+++ b/docs/topics/telnetconsole.rst@@ -151,6 +151,33 @@ To stop::>>> engine.stop()Connection closed by foreign host.++.. _telnet-concurrency:++Change concurrency and delays+-----------------------------++The downloader checks its concurrency and delay attributes for every request+it sends, so you can speed a crawl up or slow it down while it runs::++ telnet localhost 6023+ >>> downloader = engine.downloader+ >>> downloader.total_concurrency = 32+ >>> for slot in downloader.slots.values():+ ... slot.concurrency = 4+ ... slot.delay = 0.5+ ...+ >>>++``total_concurrency`` is :setting:`CONCURRENT_REQUESTS`, and ``slots`` maps the+name of every :setting:`download slot <DOWNLOAD_SLOTS>` in use to an object+whose ``concurrency`` and ``delay`` come from+:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (or+:setting:`CONCURRENT_REQUESTS_PER_IP`) and :setting:`DOWNLOAD_DELAY`. A slot+created later, e.g. for a domain not visited yet, starts from those settings+again, and :ref:`AutoThrottle <topics-autothrottle>` recalculates ``delay``+after every response.+.. skip: endTelnet Console signals