Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased
Adrián Chaves8a2768561fad
diff --cc docs/topics/coroutines.rstindex 67f1a4098,0904637b0..067a97a55--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@@ -15,11 -15,16 +15,11 @@@ Supported callableThe following callables may be defined as coroutines using ``async def``, andhence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- - :class:`~scrapy.http.Request` callbacks.+ - :class:`~scrapy.Request` callbacks.- .. note:: The callback output is not processed until the whole callback- finishes.-- As a side effect, if the callback raises an exception, none of its- output is processed.-- This is a known caveat of the current implementation that we aim to- address in a future version of Scrapy.+ .. versionchanged:: VERSION+ Output of async callbacks is now processed asynchronously instead of collecting+ all of it first.- The :meth:`process_item` method of:ref:`item pipelines <topics-item-pipeline>`.diff --cc docs/topics/spider-middleware.rstindex 7c0d4a635,f0158dc41..ece819124--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@@ -106,22 -102,19 +106,22 @@@ object gives you access, for example, tit has processed the response.:meth:`process_spider_output` must return an iterable of- :class:`~scrapy.http.Request` objects and :ref:`item objects- :class:`~scrapy.Request` objects and :ref:`item object++ :class:`~scrapy.Request` objects and :ref:`item objects<topics-items>`.+ .. note:: When defined as a :ref:`coroutine <async>`, this method needs+ to be an async generator, not just return an iterable.+:param response: the response which generated this output from thespider:type response: :class:`~scrapy.http.Response` object:param result: the result returned by the spider- :type result: an iterable of :class:`~scrapy.http.Request` objects and+ :type result: an iterable of :class:`~scrapy.Request` objects and- :ref:`item object <topics-items>`+ :ref:`item objects <topics-items>`:param spider: the spider whose result is being processed- :type spider: :class:`~scrapy.spiders.Spider` object+ :type spider: :class:`~scrapy.Spider` object.. method:: process_spider_exception(response, exception, spider)@@@ -129,7 -122,7 +129,7 @@@method (from a previous spider middleware) raises an exception.:meth:`process_spider_exception` should return either ``None`` or an- iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects- iterable of :class:`~scrapy.Request` objects and :ref:`item object++ iterable of :class:`~scrapy.Request` objects and :ref:`item objects<topics-items>`.If it returns ``None``, Scrapy will continue processing this exception,