Scrapy, a fast high-level web crawling & scraping framework for Python.
Document how blocking callback code inflates download latency (#8160)
Adrian381003e2e7f3
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rstindex ed981bcd4..3040bb283 100644--- a/docs/topics/optimize.rst+++ b/docs/topics/optimize.rst@@ -193,6 +193,46 @@ Pushed far enough, they turn memory or disk into your new bottleneck, which iswhy :ref:`optimize-memory` recommends the reverse of the last point.+.. _optimize-blocking:++Keeping the event loop free+===========================++Your callbacks, :ref:`spider middlewares <topics-spider-middleware>`,+:ref:`downloader middlewares <topics-downloader-middleware>` and :ref:`item+pipelines <topics-item-pipeline>` share a thread with the event loop. While any+of them runs, Scrapy neither sends requests nor reads responses, so every+download in flight waits for it.++That shows up as a :ref:`download latency <download-latency>` that grows+through the crawl, often into tens of seconds, while the target website answers+as fast as ever: the latency of a response covers the time until Scrapy gets+around to reading it, and every pending callback runs first.++Move the slow code to a thread so that downloads continue while it runs:++.. code-block:: python++ from scrapy import Spider+ from scrapy.utils.asyncio import run_in_thread+++ class SlowParseSpider(Spider):+ name = "slow_parse"+ start_urls = ["https://toscrape.com"]++ async def parse(self, response):+ yield await run_in_thread(self.extract_item, response)++ def extract_item(self, response):+ # CPU-heavy parsing+ ...++A thread does not give CPU-bound Python code more CPU to work with, since it+still competes for the GIL; it only keeps that code from holding up downloads.+See :ref:`distributed-crawls` to use more than one core.++.. _optimize-resources:Lowering resource usagediff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 7a30cbb1d..aa08a0a14 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -926,9 +926,11 @@ download_latency----------------The amount of time spent to fetch the response, since the request has been-started, i.e. HTTP message sent over the network. This meta key only becomes-available when the response has been downloaded. While most other meta keys are-used to control Scrapy behavior, this one is supposed to be read-only.+started, i.e. HTTP message sent over the network. It covers the time until+Scrapy reads the response, which your own code can delay, see+:ref:`optimize-blocking`. This meta key only becomes available when the+response has been downloaded. While most other meta keys are used to control+Scrapy behavior, this one is supposed to be read-only... reqmeta:: download_fail_on_dataloss