Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased

Adrián Chaves8a2768561fad
diff --cc docs/topics/coroutines.rst
index 67f1a4098,0904637b0..067a97a55
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@@ -15,11 -15,16 +15,11 @@@ Supported callable
The following callables may be defined as coroutines using ``async def``, and
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- - :class:`~scrapy.http.Request` callbacks.
+ - :class:`~scrapy.Request` callbacks.
- .. note:: The callback output is not processed until the whole callback
- finishes.
-
- As a side effect, if the callback raises an exception, none of its
- output is processed.
-
- This is a known caveat of the current implementation that we aim to
- address in a future version of Scrapy.
+ .. versionchanged:: VERSION
+ Output of async callbacks is now processed asynchronously instead of collecting
+ all of it first.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
diff --cc docs/topics/spider-middleware.rst
index 7c0d4a635,f0158dc41..ece819124
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@@ -106,22 -102,19 +106,22 @@@ object gives you access, for example, t
it has processed the response.
:meth:`process_spider_output` must return an iterable of
- :class:`~scrapy.http.Request` objects and :ref:`item objects
- :class:`~scrapy.Request` objects and :ref:`item object
++ :class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
+ .. note:: When defined as a :ref:`coroutine <async>`, this method needs
+ to be an async generator, not just return an iterable.
+
:param response: the response which generated this output from the
spider
:type response: :class:`~scrapy.http.Response` object
:param result: the result returned by the spider
- :type result: an iterable of :class:`~scrapy.http.Request` objects and
+ :type result: an iterable of :class:`~scrapy.Request` objects and
- :ref:`item object <topics-items>`
+ :ref:`item objects <topics-items>`
:param spider: the spider whose result is being processed
- :type spider: :class:`~scrapy.spiders.Spider` object
+ :type spider: :class:`~scrapy.Spider` object
.. method:: process_spider_exception(response, exception, spider)
@@@ -129,7 -122,7 +129,7 @@@
method (from a previous spider middleware) raises an exception.
:meth:`process_spider_exception` should return either ``None`` or an
- iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects
- iterable of :class:`~scrapy.Request` objects and :ref:`item object
++ iterable of :class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
If it returns ``None``, Scrapy will continue processing this exception,