Scrapy, a fast high-level web crawling & scraping framework for Python.
Add a docs section on using download_async() from a downloader middleware (#7872)
Adrianba29ce8d3bca
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 2d6ba0cf3..965108214 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -156,6 +156,61 @@ defines one or more of these methods::param exception: the raised exception:type exception: an ``Exception`` object+.. _mw-download:++Downloading a request from a downloader middleware+==================================================++A downloader middleware can download a request of its own while it processes+another one, e.g. to fetch something that the request it is processing needs.+The built-in :ref:`robots.txt middleware <topics-dlmw-robots>` does that: it+holds each request while it downloads the ``robots.txt`` file of its website.++Use :meth:`crawler.engine.download_async()+<scrapy.core.engine.ExecutionEngine.download_async>` for that:++.. code-block:: python++ from scrapy import Request+ from scrapy.http.request import NO_CALLBACK+++ class TokenMiddleware:+ def __init__(self, crawler):+ self.crawler = crawler+ self.token = None++ @classmethod+ def from_crawler(cls, crawler):+ return cls(crawler)++ async def process_request(self, request):+ if request.meta.get("dont_obey_robotstxt"):+ return+ if self.token is None:+ response = await self.crawler.engine.download_async(+ Request(+ "https://example.com/token",+ callback=NO_CALLBACK,+ meta={"dont_obey_robotstxt": True},+ )+ )+ self.token = response.text+ request.headers["Authorization"] = self.token++Requests that you download this way go through the downloader middleware chain+as well, including your own middleware and the :ref:`robots.txt middleware+<topics-dlmw-robots>`, which holds a request until the ``robots.txt`` file of+its website arrives. Be careful not to introduce deadlocks: a request that you+download must not end up waiting for the request that is waiting for it. Hence+:reqmeta:`dont_obey_robotstxt` above, which makes both middlewares let the token+request through.++While the first token response is in transit, ``process_request`` runs for other+requests as well, and the middleware above downloads a token for each of them.+Cache the task that downloads the token, and not only its result, to download+the token only once.+.. _topics-downloader-middleware-ref:Built-in downloader middleware reference