Scrapy, a fast high-level web crawling & scraping framework for Python.
Update docs.
Andrey Rakhmatullin849472535ef9
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 9b50d9312..6a39dcb5e 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -108,3 +108,68 @@ Common use cases for asynchronous code include::ref:`the screenshot pipeline example<ScreenshotPipeline>`)... _aio-libs: https://github.com/aio-libs++.. _async-spider-middlewares:++Asynchronous spider middlewares+===============================++.. versionadded:: VERSION+.. note:: This currently applies to+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`.++Middleware methods discussed here can take and return async iterables. They can+return the same type of iterable or they can take a normal one and return an+async one. If such method needs to return an async iterable it must be an async+generator, not just a coroutine that returns an iterable.++.. autofunction:: scrapy.utils.asyncgen.as_async_generator++In the simplest form that supports both sync and async input it can be written+like this::++ from scrapy.utils.asyncgen import as_async_generator++ class ProcessSpiderOutputAsyncGenMiddleware:+ async def process_spider_output(self, response, result, spider):+ async for r in as_async_generator(result):+ # ... do something with r+ yield r++If the middleware input (the callback result for ``process_spider_output``) is+an async iterable, all middlewares that process it must support it. The+built-in ones do, but the ones in your project and 3rd-party ones will need to+be updated to support it, as the code that expects a normal iterable will break+on an async one. If these middlewares receive an async iterable, they must+return one as well. On the other hand, if they receive a normal iterable, they+shouldn't break and ideally should return a normal iterable too. There can be+several possible implementations of this.++The simplest one, always converting normal iterables to async ones, is provided+above. Because a result of a middleware method is passed to the same method of+the next middleware, it's only possible to mix middlewares with synchronous and+asynchronous implementations of the same method if all synchronous ones are+called first (which isn't always possible).++Another option is to make separate methods for normal and async iterables and+choose one at run time::++ from inspect import isasyncgen++ class ProcessSpiderOutputAsyncGenMiddleware:+ def _normal_process_spider_output(self, response, result, spider):+ # ... do something with normal result++ async def _async_process_spider_output(self, response, result, spider):+ # ... do the same with async result++ def process_spider_output(self, response, result, spider):+ if isasyncgen(result):+ return self._async_process_spider_output(self, response, result, spider)+ else:+ return self._normal_process_spider_output(self, response, result, spider)++If you are writing a middleware that you intend to publish or to use in many+projects, this is likely the best way to implement it. It may be possible to+extract common code from both methods to reduce code duplication, as in the+simplest case the only difference between them will be ``for`` vs ``async for``.diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex d09693c16..d87b89292 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -105,18 +105,20 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.This method is called with the results returned from the Spider, afterit has processed the response.- :meth:`process_spider_output` must return an iterable (normal or- asynchronous) of :class:`~scrapy.http.Request` objects and- :ref:`item objects <topics-items>`.+ :meth:`process_spider_output` must return an iterable of+ :class:`~scrapy.http.Request` objects and :ref:`item objects+ <topics-items>`.++ .. note:: When defined as a :ref:`coroutine <async>`, this method needs+ to be an async generator, not just return an iterable.:param response: the response which generated this output from thespider:type response: :class:`~scrapy.http.Response` object:param result: the result returned by the spider- :type result: an iterable (normal or asynchronous) of- :class:`~scrapy.http.Request` objects and :ref:`item objects- <topics-items>`+ :type result: an iterable of :class:`~scrapy.http.Request` objects and+ :ref:`item objects <topics-items>`:param spider: the spider whose result is being processed:type spider: :class:`~scrapy.spiders.Spider` objectdiff --git a/scrapy/utils/asyncgen.py b/scrapy/utils/asyncgen.pyindex a79552f76..6c0bb1d10 100644--- a/scrapy/utils/asyncgen.py+++ b/scrapy/utils/asyncgen.py@@ -11,6 +11,7 @@ async def collect_asyncgen(result):async def as_async_generator(it):+ """ Wraps an iterator (sync or async) into an async generator. """if isinstance(it, collections.abc.AsyncIterator):async for r in it:yield r