Scrapy, a fast high-level web crawling & scraping framework for Python.
Add/improve docs.
Andrey Rakhmatullina97dc55c714d
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 4addaa178..18712c928 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -38,7 +38,6 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class touse it instead of the default asyncio event loop.-.. _asyncio-await-dfd:Awaiting on Deferredsdiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex 279632653..9b50d9312 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -17,6 +17,10 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :class:`~scrapy.http.Request` callbacks.+ .. versionchanged:: VERSION+ Output of async callbacks is now processed asynchronously instead of collecting+ all of it first.+- The :meth:`process_item` method of:ref:`item pipelines <topics-item-pipeline>`.@@ -30,6 +34,13 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :ref:`Signal handlers that support deferreds <signal-deferred>`.+- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`+ method of :ref:`spider middlewares <custom-spider-middleware>`.++ .. versionadded:: VERSION+ .. note:: This method needs to be an async generator, not just a coroutine that+ returns an iterable.+Usage=====@@ -76,7 +87,7 @@ This means you can use many useful Python libraries providing such code::async def parse_with_asyncio(self, response):async with aiohttp.ClientSession() as session:async with session.get('https://additional.url') as additional_response:- additional_data = await r.text()+ additional_data = await additional_response.text()# ... use response and additional_data to yield items and requests.. note:: Many libraries that use coroutines, such as `aio-libs`_, require thediff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex fc114a63f..d09693c16 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -98,20 +98,25 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`... method:: process_spider_output(response, result, spider)+ .. versionchanged:: VERSION+ Since VERSION this can take and return an :term:`python:asynchronous+ iterable`.+This method is called with the results returned from the Spider, afterit has processed the response.- :meth:`process_spider_output` must return an iterable of- :class:`~scrapy.http.Request` objects and :ref:`item object- <topics-items>`.+ :meth:`process_spider_output` must return an iterable (normal or+ asynchronous) of :class:`~scrapy.http.Request` objects and+ :ref:`item objects <topics-items>`.:param response: the response which generated this output from thespider:type response: :class:`~scrapy.http.Response` object:param result: the result returned by the spider- :type result: an iterable of :class:`~scrapy.http.Request` objects and- :ref:`item object <topics-items>`+ :type result: an iterable (normal or asynchronous) of+ :class:`~scrapy.http.Request` objects and :ref:`item objects+ <topics-items>`:param spider: the spider whose result is being processed:type spider: :class:`~scrapy.spiders.Spider` object@@ -122,7 +127,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.method (from a previous spider middleware) raises an exception.:meth:`process_spider_exception` should return either ``None`` or an- iterable of :class:`~scrapy.http.Request` objects and :ref:`item object+ iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects<topics-items>`.If it returns ``None``, Scrapy will continue processing this exception,