Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document how blocking callback code inflates download latency (#8160)

Adrian381003e2e7f3
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst
index ed981bcd4..3040bb283 100644
--- a/docs/topics/optimize.rst
+++ b/docs/topics/optimize.rst
@@ -193,6 +193,46 @@ Pushed far enough, they turn memory or disk into your new bottleneck, which is
why :ref:`optimize-memory` recommends the reverse of the last point.
+.. _optimize-blocking:
+
+Keeping the event loop free
+===========================
+
+Your callbacks, :ref:`spider middlewares <topics-spider-middleware>`,
+:ref:`downloader middlewares <topics-downloader-middleware>` and :ref:`item
+pipelines <topics-item-pipeline>` share a thread with the event loop. While any
+of them runs, Scrapy neither sends requests nor reads responses, so every
+download in flight waits for it.
+
+That shows up as a :ref:`download latency <download-latency>` that grows
+through the crawl, often into tens of seconds, while the target website answers
+as fast as ever: the latency of a response covers the time until Scrapy gets
+around to reading it, and every pending callback runs first.
+
+Move the slow code to a thread so that downloads continue while it runs:
+
+.. code-block:: python
+
+ from scrapy import Spider
+ from scrapy.utils.asyncio import run_in_thread
+
+
+ class SlowParseSpider(Spider):
+ name = "slow_parse"
+ start_urls = ["https://toscrape.com"]
+
+ async def parse(self, response):
+ yield await run_in_thread(self.extract_item, response)
+
+ def extract_item(self, response):
+ # CPU-heavy parsing
+ ...
+
+A thread does not give CPU-bound Python code more CPU to work with, since it
+still competes for the GIL; it only keeps that code from holding up downloads.
+See :ref:`distributed-crawls` to use more than one core.
+
+
.. _optimize-resources:
Lowering resource usage
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 7a30cbb1d..aa08a0a14 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -926,9 +926,11 @@ download_latency
----------------
The amount of time spent to fetch the response, since the request has been
-started, i.e. HTTP message sent over the network. This meta key only becomes
-available when the response has been downloaded. While most other meta keys are
-used to control Scrapy behavior, this one is supposed to be read-only.
+started, i.e. HTTP message sent over the network. It covers the time until
+Scrapy reads the response, which your own code can delay, see
+:ref:`optimize-blocking`. This meta key only becomes available when the
+response has been downloaded. While most other meta keys are used to control
+Scrapy behavior, this one is supposed to be read-only.
.. reqmeta:: download_fail_on_dataloss