Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add/improve docs.

Andrey Rakhmatullina97dc55c714d
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index 4addaa178..18712c928 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -38,7 +38,6 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the
:setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to
use it instead of the default asyncio event loop.
-
.. _asyncio-await-dfd:
Awaiting on Deferreds
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 279632653..9b50d9312 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -17,6 +17,10 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.http.Request` callbacks.
+ .. versionchanged:: VERSION
+ Output of async callbacks is now processed asynchronously instead of collecting
+ all of it first.
+
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@@ -30,6 +34,13 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
+- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
+ method of :ref:`spider middlewares <custom-spider-middleware>`.
+
+ .. versionadded:: VERSION
+ .. note:: This method needs to be an async generator, not just a coroutine that
+ returns an iterable.
+
Usage
=====
@@ -76,7 +87,7 @@ This means you can use many useful Python libraries providing such code::
async def parse_with_asyncio(self, response):
async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response:
- additional_data = await r.text()
+ additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index fc114a63f..d09693c16 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -98,20 +98,25 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. method:: process_spider_output(response, result, spider)
+ .. versionchanged:: VERSION
+ Since VERSION this can take and return an :term:`python:asynchronous
+ iterable`.
+
This method is called with the results returned from the Spider, after
it has processed the response.
- :meth:`process_spider_output` must return an iterable of
- :class:`~scrapy.http.Request` objects and :ref:`item object
- <topics-items>`.
+ :meth:`process_spider_output` must return an iterable (normal or
+ asynchronous) of :class:`~scrapy.http.Request` objects and
+ :ref:`item objects <topics-items>`.
:param response: the response which generated this output from the
spider
:type response: :class:`~scrapy.http.Response` object
:param result: the result returned by the spider
- :type result: an iterable of :class:`~scrapy.http.Request` objects and
- :ref:`item object <topics-items>`
+ :type result: an iterable (normal or asynchronous) of
+ :class:`~scrapy.http.Request` objects and :ref:`item objects
+ <topics-items>`
:param spider: the spider whose result is being processed
:type spider: :class:`~scrapy.spiders.Spider` object
@@ -122,7 +127,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
method (from a previous spider middleware) raises an exception.
:meth:`process_spider_exception` should return either ``None`` or an
- iterable of :class:`~scrapy.http.Request` objects and :ref:`item object
+ iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects
<topics-items>`.
If it returns ``None``, Scrapy will continue processing this exception,