Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Refactor the asynchronous process_spider_output documentation

Adrián Chavesfd08bb6cd99f
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 073b6bd9a..361cd5e60 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -17,9 +17,12 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.Request` callbacks.
+ If you are using any custom or third-party :ref:`spider middleware
+ <topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
+
.. versionchanged:: VERSION
- Output of async callbacks is now processed asynchronously instead of collecting
- all of it first.
+ Output of async callbacks is now processed asynchronously instead of
+ collecting all of it first.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@@ -34,18 +37,26 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
-- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
- method of :ref:`spider middlewares <custom-spider-middleware>`. See
- :ref:`async-spider-middlewares`.
+- The
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
+ method of :ref:`spider middlewares <topics-spider-middleware>`.
+
+ It must be defined as an :term:`asynchronous generator`. The input
+ ``result`` parameter is an :term:`asynchronous iterable`.
+
+ See also :ref:`sync-async-spider-middleware` and
+ :ref:`universal-spider-middleware`.
.. versionadded:: VERSION
-Usage
-=====
+General usage
+=============
+
+There are several use cases for coroutines in Scrapy.
-There are several use cases for coroutines in Scrapy. Code that would
-return Deferreds when written for previous Scrapy versions, such as downloader
-middlewares and signal handlers, can be rewritten to be shorter and cleaner::
+Code that would return Deferreds when written for previous Scrapy versions,
+such as downloader middlewares and signal handlers, can be rewritten to be
+shorter and cleaner::
from itemadapter import ItemAdapter
@@ -110,46 +121,73 @@ Common use cases for asynchronous code include:
.. _aio-libs: https://github.com/aio-libs
-.. _async-spider-middlewares:
-Asynchronous spider middlewares
-===============================
+.. _sync-async-spider-middleware:
+
+Mixing synchronous and asynchronous spider middlewares
+======================================================
+
+.. versionadded:: VERSION
+
+The output of a :class:`~scrapy.Request` callback is passed as the ``result``
+parameter to the
+:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
+of the first :ref:`spider middleware <topics-spider-middleware>` from the
+:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.
+Then the output of that ``process_spider_output`` method is passed to the
+``process_spider_output`` method of the next spider middleware, and so on for
+every active spider middleware.
+
+Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods
+in this chain of calls.
+
+However, if any of the ``process_spider_output`` methods is defined as a
+synchronous method, and the previous ``Request`` callback or
+``process_spider_output`` method is a coroutine, there are some drawbacks to
+the asynchronous-to-synchronous conversion that Scrapy does so that the
+synchronous ``process_spider_output`` method gets a synchronous iterable as its
+``result`` parameter:
+
+- The whole output of the previous ``Request`` callback or
+ ``process_spider_output`` method is awaited at this point.
+
+- If an exception raises while awaiting the output of the previous
+ ``Request`` callback or ``process_spider_output`` method, none of that
+ output will be processed.
+
+Asynchronous-to-synchronous conversions are supported for backward
+compatibility, but they are deprecated and will stop working in a future
+version of Scrapy.
+
+To avoid asynchronous-to-synchronous conversion, when defining ``Request``
+callbacks as coroutine methods or when using spider middlewares whose
+``process_spider_output`` method is an :term:`asynchronous generator`, all
+active spider middlewares must either have their ``process_spider_output``
+method defined as an asynchronous generator or :ref:`define a
+process_spider_output_async method <universal-spider-middleware>`.
+
+.. note:: When using third-party spider middlewares that only define a
+ synchronous ``process_spider_output`` method, consider
+ :ref:`making them universal <universal-spider-middleware>` through
+ :ref:`subclassing <tut-inheritance>`.
+
+
+.. _universal-spider-middleware:
+
+Universal spider middleware
+===========================
.. versionadded:: VERSION
-.. note:: This currently applies to
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`.
- In the future it will also apply to
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`.
-
-Middleware methods discussed here can take and return async iterables. They can
-return the same type of iterable or they can take a normal one and return an
-async one. If such method needs to return an async iterable it must be an async
-generator, not just a coroutine that returns an iterable.
-
-As the result of a middleware method is passed to the same method of the next
-middleware, it needs to be adapted if the second method expects a different
-type. Scrapy will do this transparently:
-
-* A normal iterable is wrapped into an async one which shouldn't cause any side
- effects.
-* An async iterable is downgraded to a normal one by waiting until all results
- are available and wrapping them in a normal iterable. This is problematic
- because it pauses the normal middleware processing for this iterable and
- because all results can be skipped if exceptions are raised during
- processing. This case emits a warning and will be deprecated and then removed
- in a later Scrapy version.
-* Async iterables returned from
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
- won't be downgraded, an exception will be raised if that is needed.
-
-As downgrading is undesirable, here is the proposed way to avoid it. If all
-middlewares, including 3rd-party ones, support async iterables as input, no
-downgrading will happen. But removing normal iterable support (making the
-method a coroutine) from a middleware published as a separate project or used
-internally in projects for older Scrapy versions breaks backwards
-compatibility. So, as an interim measure (it will be deprecated and then
-removed in a later Scrapy version), a middleware can provide both sync and
-async methods in the following form::
+
+To allow writing a spider middleware that supports asynchronous execution of
+its ``process_spider_output`` method in Scrapy VERSION and later (avoiding
+:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
+while maintaining support for older Scrapy versions, you may define
+``process_spider_output`` as a synchronous method and define an
+:term:`asynchronous generator` version of that method with an alternative name:
+``process_spider_output_async``.
+
+For example::
class UniversalSpiderMiddleware:
def process_spider_output(self, response, result, spider):
@@ -162,28 +200,13 @@ async methods in the following form::
# ... do something with r
yield r
-In this case normal and async iterables will be passed to the respective
-methods without any wrapping or downgrading, and in older versions of Scrapy
-the coroutine method will just be ignored. When the backwards compatibility is
-no longer needed the non-coroutine method can be dropped and the coroutine one
-renamed to the normal name. It may be possible to extract common code from both
-methods to reduce code duplication, as in the simplest case the only difference
-between them will be ``for`` vs ``async for``.
-
-So, to recap:
-
-* If you don't intend to use async callbacks or middlewares containing async
- code in your project, nothing should change for you yet. At some point in the
- future some of the 3rd-party middlewares you use may drop backwards
- compatibility, which shouldn't lead to immediate problems but may be a sign
- to start converting your code to ``async def`` too.
-* If you maintain a middleware that can be used with projects you can't control
- (e.g. one you published for other people to use, or one that needs to support
- some old project that can't be modernized), we recommend adding a
- ``process_spider_output_async`` method so that the amount of unnecessary
- iterable conversions is reduced but no compatibility is broken.
-* If you use async callbacks, try to make sure all middlewares support them.
- Note that you can modernize 3rd-party middlewares by subclassing them.
-* If you want to write and publish a middleware that requires async code, you
- should write in the docs that the minimum support Scrapy version is VERSION
- (maybe even check this at the run time, using :attr:`scrapy.__version__`).
+.. note:: This is an interim measure to allow, for a time, to write code that
+ works in Scrapy VERSION and later without requiring
+ asynchronous-to-synchronous conversions, and works in earlier Scrapy
+ versions as well.
+
+ In some future version of Scrapy, however, this feature will be
+ deprecated and, eventually, in a later version of Scrapy, this
+ feature will be removed, and all spider middlewares will be expected
+ to define their ``process_spider_output`` method as an asynchronous
+ generator.
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index edfc2e4bb..787545ed2 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -98,10 +98,6 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. method:: process_spider_output(response, result, spider)
- .. versionchanged:: VERSION
- Since VERSION this can take and return an :term:`python:asynchronous
- iterable`.
-
This method is called with the results returned from the Spider, after
it has processed the response.
@@ -109,8 +105,15 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
- .. note:: When defined as a :ref:`coroutine <async>`, this method needs
- to be an async generator, not just return an iterable.
+ .. versionchanged:: VERSION
+ This method may be defined as an :term:`asynchronous generator`, in
+ which case ``result`` is an :term:`asynchronous iterable`.
+
+ Consider defining this method as an :term:`asynchronous generator`,
+ which will be a requirement in a future version of Scrapy. However, if
+ you wish your spider middleware to work with Scrapy versions earlier
+ than Scrapy VERSION, :ref:`make your spider middleware universal
+ <universal-spider-middleware>` instead.
:param response: the response which generated this output from the
spider
@@ -127,10 +130,9 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. versionadded:: VERSION
- If exists, this methid will be called instead of
- :meth:`process_spider_output` when ``result`` is an async iterable.
- If this method exists, it must be a coroutine while
- :meth:`process_spider_output` must not be a coroutine.
+ If defined, this method must be an :term:`asynchronous generator`,
+ which will be called instead of :meth:`process_spider_output` if
+ ``result`` is an :term:`asynchronous iterable`.
.. method:: process_spider_exception(response, exception, spider)