Scrapy, a fast high-level web crawling & scraping framework for Python.
Remove support for sync process_spider_output(). (#7504)
Andrey Rakhmatullina8a8f20d9c49
diff --git a/docs/news.rst b/docs/news.rstindex 5eef10d84..3173dfbc1 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -1380,8 +1380,8 @@ Highlights:- Added the :reqmeta:`allow_offsite` request meta key-- :ref:`Spider middlewares that don't support asynchronous spider output- <sync-async-spider-middleware>` are deprecated+- Spider middlewares that don't support asynchronous spider output are+ deprecated- Added a base class for :ref:`universal spider middlewares<universal-spider-middleware>`@@ -1519,13 +1519,11 @@ Deprecations``start_queue_cls`` parameter.(:issue:`6752`)-- :ref:`Spider middlewares that don't support asynchronous spider output- <sync-async-spider-middleware>` are deprecated. The async iterable- downgrading feature, needed for using such middlewares with asynchronous- callbacks and with other spider middlewares that produce asynchronous- iterables, is also deprecated. Please update all such middlewares to- support asynchronous spider output.- (:issue:`6664`)+- Spider middlewares that don't support asynchronous spider output are+ deprecated. The async iterable downgrading feature, needed for using such+ middlewares with asynchronous callbacks and with other spider middlewares+ that produce asynchronous iterables, is also deprecated. Please update all+ such middlewares to support asynchronous spider output. (:issue:`6664`)- Functions that were imported from :mod:`w3lib.url` and re-exported in:mod:`scrapy.utils.url` are now deprecated, you should import them from@@ -1784,9 +1782,8 @@ Documentation- Documented the setting values set in the default project template.(:issue:`6762`, :issue:`6775`)-- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous- iterable support in spider middlewares.- (:issue:`6688`)+- Improved the docs about asynchronous iterable support in spider+ middlewares. (:issue:`6688`)- Improved the :ref:`docs <coroutine-deferred-apis>` about using:class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-baseddiff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex ea3a60b4c..ba68f0dbc 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -23,9 +23,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :class:`~scrapy.Request` callbacks.- If you are using any custom or third-party :ref:`spider middleware- <topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.-- The :meth:`process_item` method of:ref:`item pipelines <topics-item-pipeline>`.@@ -39,13 +36,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- The:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`- method of :ref:`spider middlewares <topics-spider-middleware>`.-- If defined as a coroutine, it must be an :term:`asynchronous generator`.- The input ``result`` parameter is an :term:`asynchronous iterable`.-- See also :ref:`sync-async-spider-middleware` and- :ref:`universal-spider-middleware`.+ method of :ref:`spider middlewares <topics-spider-middleware>`, which+ *must* be defined as an :term:`asynchronous generator` except in+ :ref:`universal spider middlewares <universal-spider-middleware>`.- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` methodof :ref:`spider middlewares <custom-spider-middleware>`, which *must* be@@ -277,136 +270,3 @@ You can also send multiple requests in parallel:"price": responses[0][1].css(".price::text").get(),"price2": responses[1][1].css(".color::text").get(),}---.. _sync-async-spider-middleware:--Mixing synchronous and asynchronous spider middlewares-======================================================--The output of a :class:`~scrapy.Request` callback is passed as the ``result``-parameter to the-:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method-of the first :ref:`spider middleware <topics-spider-middleware>` from the-:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.-Then the output of that ``process_spider_output`` method is passed to the-``process_spider_output`` method of the next spider middleware, and so on for-every active spider middleware.--Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods-in this chain of calls.--However, if any of the ``process_spider_output`` methods is defined as a-synchronous method, and the previous ``Request`` callback or-``process_spider_output`` method is a coroutine, there are some drawbacks to-the asynchronous-to-synchronous conversion that Scrapy does so that the-synchronous ``process_spider_output`` method gets a synchronous iterable as its-``result`` parameter:--- The whole output of the previous ``Request`` callback or- ``process_spider_output`` method is awaited at this point.--- If an exception raises while awaiting the output of the previous- ``Request`` callback or ``process_spider_output`` method, none of that- output will be processed.-- This contrasts with the regular behavior, where all items yielded before- an exception raises are processed.--Asynchronous-to-synchronous conversions are supported for backward-compatibility, but they are deprecated and will stop working in a future-version of Scrapy.--To avoid asynchronous-to-synchronous conversions, when defining ``Request``-callbacks as coroutine methods or when using spider middlewares whose-``process_spider_output`` method is an :term:`asynchronous generator`, all-active spider middlewares must either have their ``process_spider_output``-method defined as an asynchronous generator or :ref:`define a-process_spider_output_async method <universal-spider-middleware>`.--.. _sync-async-spider-middleware-users:--For middleware users-----------------------If you have asynchronous callbacks or use asynchronous-only spider middlewares-you should make sure the asynchronous-to-synchronous conversions-:ref:`described above <sync-async-spider-middleware>` don't happen. To do this,-make sure all spider middlewares you use support asynchronous spider output.-Even if you don't have asynchronous callbacks and don't use asynchronous-only-spider middlewares in your project, it's still a good idea to make sure all-middlewares you use support asynchronous spider output, so that it will be easy-to start using asynchronous callbacks in the future. Because of this, Scrapy-logs a warning when it detects a synchronous-only spider middleware.--If you want to update middlewares you wrote, see the :ref:`following section-<sync-async-spider-middleware-authors>`. If you have 3rd-party middlewares that-aren't yet updated by their authors, you can :ref:`subclass <tut-inheritance>`-them to make them :ref:`universal <universal-spider-middleware>` and use the-subclasses in your projects.--.. _sync-async-spider-middleware-authors:--For middleware authors-------------------------If you have a spider middleware that defines a synchronous-``process_spider_output`` method, you should update it to support asynchronous-spider output for :ref:`better compatibility <sync-async-spider-middleware>`,-even if you don't yet use it with asynchronous callbacks, especially if you-publish this middleware for other people to use. You have two options for this:--1. Make the middleware asynchronous, by making the ``process_spider_output``- method an :term:`asynchronous generator`.-2. Make the middleware universal, as described in the :ref:`next section- <universal-spider-middleware>`.--If your middleware won't be used in projects with synchronous-only middlewares,-e.g. because it's an internal middleware and you know that all other-middlewares in your projects are already updated, it's safe to choose the first-option. Otherwise, it's better to choose the second option.--.. _universal-spider-middleware:--Universal spider middlewares-------------------------------To allow writing a spider middleware that supports asynchronous execution of-its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding-:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)-while maintaining support for older Scrapy versions, you may define-``process_spider_output`` as a synchronous method and define an-:term:`asynchronous generator` version of that method with an alternative name:-``process_spider_output_async``.--For example:--.. code-block:: python-- class UniversalSpiderMiddleware:- def process_spider_output(self, response, result):- for r in result:- # ... do something with r- yield r-- async def process_spider_output_async(self, response, result):- async for r in result:- # ... do something with r- yield r--.. note:: This is an interim measure to allow, for a time, to write code that- works in Scrapy 2.7 and later without requiring- asynchronous-to-synchronous conversions, and works in earlier Scrapy- versions as well.-- In some future version of Scrapy, however, this feature will be- deprecated and, eventually, in a later version of Scrapy, this- feature will be removed, and all spider middlewares will be expected- to define their ``process_spider_output`` method as an asynchronous- generator.--Since 2.13.0, Scrapy provides a base class,-:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements-the ``process_spider_output()`` and ``process_spider_output_async()`` methods,-so instead of duplicating the processing code you can override the-``get_processed_request()`` and/or the ``get_processed_item()`` method.diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 8b000697a..820d5910c 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -117,36 +117,28 @@ one or more of these methods::type response: :class:`~scrapy.http.Response` object.. method:: process_spider_output(response, result)+ :async:- This method is called with the results returned from the Spider, after- it has processed the response.-- :meth:`process_spider_output` must return an iterable of- :class:`~scrapy.Request` objects and :ref:`item objects- <topics-items>`.+ This method is an :term:`asynchronous generator` called with the+ results from the spider after the spider has processed the response.- Consider defining this method as an :term:`asynchronous generator`,- which will be a requirement in a future version of Scrapy. However, if- you plan on sharing your spider middleware with other people, consider- either :ref:`enforcing Scrapy 2.7 <enforce-component-requirements>`- as a minimum requirement of your spider middleware, or :ref:`making- your spider middleware universal <universal-spider-middleware>` so that- it works with Scrapy versions earlier than Scrapy 2.7.+ .. seealso:: :ref:`universal-spider-middleware`.:param response: the response which generated this output from thespider:type response: :class:`~scrapy.http.Response` object- :param result: the result returned by the spider- :type result: an iterable of :class:`~scrapy.Request` objects and- :ref:`item objects <topics-items>`+ :param result: the results from the spider+ :type result: an :term:`asynchronous iterable` of+ :class:`~scrapy.Request` objects and :ref:`item objects+ <topics-items>`.. method:: process_spider_output_async(response, result):async:- If defined, this method must be an :term:`asynchronous generator`,- which will be called instead of :meth:`process_spider_output` if- ``result`` is an :term:`asynchronous iterable`.+ Alternative name for :meth:`process_spider_output` used when+ implementing a :ref:`universal spider middleware+ <universal-spider-middleware>`... method:: process_spider_exception(response, exception)@@ -174,13 +166,40 @@ one or more of these methods::type exception: :exc:`Exception` object+.. _universal-spider-middleware:++Universal spider middlewares+----------------------------++In Scrapy 2.6.3 and lower, ``process_spider_output()`` must be a *synchronous*+generator.++To support those versions and higher Scrapy versions in the same middleware,+rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()`+method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define+a synchronous ``process_spider_output()`` method to be used by 2.6.3 and lower+versions.++For example:++.. code-block:: python++ class UniversalSpiderMiddleware:+ async def process_spider_output_async(self, response, result):+ async for r in result:+ # ... do something with r+ yield r++ def process_spider_output(self, response, result):+ for r in result:+ # ... do something with r+ yield r+Base class for custom spider middlewares----------------------------------------Scrapy provides a base class for custom spider middlewares. It's not required-to use it but it can help with simplifying middleware implementations and-reducing the amount of boilerplate code in :ref:`universal middlewares-<universal-spider-middleware>`.+to use it but it can help with simplifying middleware implementations... module:: scrapy.spidermiddlewares.basediff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.pyindex d48a65967..bcf38bb75 100644--- a/scrapy/core/spidermw.py+++ b/scrapy/core/spidermw.py@@ -9,29 +9,28 @@ from __future__ import annotationsimport loggingfrom collections.abc import AsyncIterator, Callable, Coroutine, Iterablefrom functools import wraps-from inspect import isasyncgenfunction, iscoroutine+from inspect import isasyncgenfunctionfrom itertools import islicefrom typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, castfrom warnings import warn-from twisted.internet.defer import Deferred, inlineCallbacksfrom twisted.python.failure import Failurefrom scrapy import Request, Spiderfrom scrapy.exceptions import ScrapyDeprecationWarning, _InvalidOutputfrom scrapy.http import Responsefrom scrapy.middleware import MiddlewareManager-from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen+from scrapy.utils.asyncgen import as_async_generatorfrom scrapy.utils.conf import build_component_listfrom scrapy.utils.defer import (_defer_sleep_async,deferred_from_coro,maybe_deferred_to_future,)-from scrapy.utils.python import MutableAsyncChain, MutableChain, global_object_name+from scrapy.utils.python import MutableAsyncChain, global_object_nameif TYPE_CHECKING:- from collections.abc import Generator+ from twisted.internet.defer import Deferredfrom scrapy.settings import BaseSettings@@ -46,10 +45,6 @@ ScrapeFunc: TypeAlias = Callable[]-def _isiterable(o: Any) -> bool:- return isinstance(o, (Iterable, AsyncIterator))--class SpiderMiddlewareManager(MiddlewareManager):component_name = "spider middleware"@@ -67,13 +62,10 @@ class SpiderMiddlewareManager(MiddlewareManager):if hasattr(mw, "process_start"):self.methods["process_start"].appendleft(mw.process_start)- process_spider_output = self._get_async_method_pair(mw, "process_spider_output")+ process_spider_output = self._get_process_spider_output(mw)self.methods["process_spider_output"].appendleft(process_spider_output)- if callable(process_spider_output):+ if process_spider_output is not None:self._check_mw_method_spider_arg(process_spider_output)- elif isinstance(process_spider_output, tuple):- for m in process_spider_output:- self._check_mw_method_spider_arg(m)process_spider_exception = getattr(mw, "process_spider_exception", None)self.methods["process_spider_exception"].appendleft(process_spider_exception)@@ -105,48 +97,7 @@ class SpiderMiddlewareManager(MiddlewareManager):return await scrape_func(Failure(), request)return await scrape_func(response, request)- def _evaluate_iterable(- self,- response: Response,- iterable: Iterable[_T] | AsyncIterator[_T],- exception_processor_index: int,- recover_to: MutableChain[_T] | MutableAsyncChain[_T],- ) -> Iterable[_T] | AsyncIterator[_T]:-- if isinstance(iterable, AsyncIterator):- return self._process_async(- response,- iterable,- exception_processor_index,- cast("MutableAsyncChain[_T]", recover_to),- )- return self._process_sync(- response,- iterable,- exception_processor_index,- cast("MutableChain[_T]", recover_to),- )-- def _process_sync(- self,- response: Response,- iterable: Iterable[_T],- exception_processor_index: int,- recover_to: MutableChain[_T],- ) -> Iterable[_T]:- try:- yield from iterable- except Exception as ex:- exception_result = cast(- "Failure | MutableChain[_T]",- self._process_spider_exception(response, ex, exception_processor_index),- )- if isinstance(exception_result, Failure):- raise- assert isinstance(recover_to, MutableChain)- recover_to.extend(exception_result)-- async def _process_async(+ async def _evaluate_iterable(self,response: Response,iterable: AsyncIterator[_T],@@ -157,13 +108,9 @@ class SpiderMiddlewareManager(MiddlewareManager):async for r in iterable:yield rexcept Exception as ex:- exception_result = cast(- "Failure | MutableAsyncChain[_T]",- self._process_spider_exception(response, ex, exception_processor_index),+ exception_result: MutableAsyncChain[_T] = self._process_spider_exception(+ response, ex, exception_processor_index)- if isinstance(exception_result, Failure):- raise- assert isinstance(recover_to, MutableAsyncChain)recover_to.extend(exception_result)def _process_spider_exception(@@ -171,7 +118,7 @@ class SpiderMiddlewareManager(MiddlewareManager):response: Response,exception: Exception,start_index: int = 0,- ) -> MutableChain[_T] | MutableAsyncChain[_T]:+ ) -> MutableAsyncChain[_T]:# don't handle _InvalidOutput exceptionif isinstance(exception, _InvalidOutput):raise exception@@ -181,28 +128,18 @@ class SpiderMiddlewareManager(MiddlewareManager):for method_index, method in enumerate(method_list, start=start_index):if method is None:continue- method = cast("Callable", method)if method in self._mw_methods_requiring_spider:result = method(response=response, exception=exception, spider=self._spider)else:result = method(response=response, exception=exception)- if _isiterable(result):+ if isinstance(result, (Iterable, AsyncIterator)):# stop exception handling by handing control over to the# process_spider_output chain if an iterable has been returned- dfd: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = (- self._process_spider_output(response, result, method_index + 1)- )- # _process_spider_output() returns a Deferred only because of downgrading so this can be- # simplified when downgrading is removed.- if dfd.called:- # the result is available immediately if _process_spider_output didn't do downgrading- return cast("MutableChain[_T] | MutableAsyncChain[_T]", dfd.result)- # we forbid waiting here because otherwise we would need to return a deferred from- # _process_spider_exception too, which complicates the architecture- msg = f"Async iterable returned from {global_object_name(method)} cannot be downgraded"- raise _InvalidOutput(msg)+ if isinstance(result, Iterable):+ result = as_async_generator(result)+ return self._process_spider_output(response, result, method_index + 1)if result is None:continuemsg = (@@ -212,124 +149,35 @@ class SpiderMiddlewareManager(MiddlewareManager):raise _InvalidOutput(msg)raise exception- # This method cannot be made async def, as _process_spider_exception relies on the Deferred result- # being available immediately which doesn't work when it's a wrapped coroutine.- # It also needs @inlineCallbacks only because of downgrading so it can be removed when downgrading is removed.- @inlineCallbacks- def _process_spider_output( # noqa: PLR0912+ def _process_spider_output(self,response: Response,- result: Iterable[_T] | AsyncIterator[_T],+ result: AsyncIterator[_T],start_index: int = 0,- ) -> Generator[Deferred[Any], Any, MutableChain[_T] | MutableAsyncChain[_T]]:+ ) -> MutableAsyncChain[_T]:# items in this iterable do not need to go through the process_spider_output# chain, they went through it already from the process_spider_exception method- recovered: MutableChain[_T] | MutableAsyncChain[_T]- last_result_is_async = isinstance(result, AsyncIterator)- recovered = MutableAsyncChain() if last_result_is_async else MutableChain()-- # There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.- # 1. def foo. Sync iterables are passed as is, async ones are downgraded.- # 2. async def foo. Sync iterables are upgraded, async ones are passed as is.- # 3. def foo + async def foo_async. Iterables are passed to the respective method.- # Storing methods and method tuples in the same list is weird but we should be able to roll this back- # when we drop this compatibility feature.-+ recovered: MutableAsyncChain[_T] = MutableAsyncChain()method_list = islice(self.methods["process_spider_output"], start_index, None)- for method_index, method_pair in enumerate(method_list, start=start_index):- if method_pair is None:+ for method_index, method in enumerate(method_list, start=start_index):+ if method is None:continue- need_upgrade = need_downgrade = False- if isinstance(method_pair, tuple):- # This tuple handling is only needed until _async compatibility methods are removed.- method_sync, method_async = method_pair- method = method_async if last_result_is_async else method_sync- else:- method = method_pair- if not last_result_is_async and isasyncgenfunction(method):- need_upgrade = True- elif last_result_is_async and not isasyncgenfunction(method):- need_downgrade = True- try:- if need_upgrade:- # Iterable -> AsyncIterator- result = as_async_generator(result)- elif need_downgrade:- logger.warning(- f"Async iterable passed to {global_object_name(method)} was"- f" downgraded to a non-async one. This is deprecated and will"- f" stop working in a future version of Scrapy. Please see"- f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"- f" for more information."- )- assert isinstance(result, AsyncIterator)- # AsyncIterator -> Iterable- result = yield deferred_from_coro(collect_asyncgen(result))- if isinstance(recovered, AsyncIterator):- recovered_collected = yield deferred_from_coro(- collect_asyncgen(recovered)- )- recovered = MutableChain(recovered_collected)- # might fail directly if the output value is not a generator- if method in self._mw_methods_requiring_spider:- result = method(- response=response, result=result, spider=self._spider- )- else:- result = method(response=response, result=result)- except Exception as ex:- exception_result: Failure | MutableChain[_T] | MutableAsyncChain[_T] = (- self._process_spider_exception(response, ex, method_index + 1)- )- if isinstance(exception_result, Failure):- raise- return exception_result- if _isiterable(result):- result = self._evaluate_iterable(- response, result, method_index + 1, recovered- )+ if method in self._mw_methods_requiring_spider:+ result = method(response=response, result=result, spider=self._spider)else:- if iscoroutine(result):- result.close() # Silence warning about not awaiting- msg = (- f"{global_object_name(method)} must be an asynchronous "- f"generator (i.e. use yield)"- )- else:- msg = (- f"{global_object_name(method)} must return an iterable, got "- f"{type(result)}"- )- raise _InvalidOutput(msg)- last_result_is_async = isinstance(result, AsyncIterator)-- if last_result_is_async:- return MutableAsyncChain(result, recovered)- return MutableChain(result, recovered) # type: ignore[arg-type]+ result = method(response=response, result=result)+ result = self._evaluate_iterable(+ response, result, method_index + 1, recovered+ )+ return MutableAsyncChain(result, recovered)async def _process_callback_output(- self,- response: Response,- result: Iterable[_T] | AsyncIterator[_T],- ) -> MutableChain[_T] | MutableAsyncChain[_T]:- recovered: MutableChain[_T] | MutableAsyncChain[_T]- if isinstance(result, AsyncIterator):- recovered = MutableAsyncChain()- else:- recovered = MutableChain()+ self, response: Response, result: AsyncIterator[_T]+ ) -> MutableAsyncChain[_T]:+ recovered: MutableAsyncChain[_T] = MutableAsyncChain()result = self._evaluate_iterable(response, result, 0, recovered)- result = await maybe_deferred_to_future(- cast(- "Deferred[Iterable[_T] | AsyncIterator[_T]]",- self._process_spider_output(response, result),- )- )- if isinstance(result, AsyncIterator):- return MutableAsyncChain(result, recovered)- if isinstance(recovered, AsyncIterator):- recovered_collected = await collect_asyncgen(recovered)- recovered = MutableChain(recovered_collected)- return MutableChain(result, recovered)+ result = self._process_spider_output(response, result)+ return MutableAsyncChain(result, recovered)def scrape_response(self,@@ -340,7 +188,7 @@ class SpiderMiddlewareManager(MiddlewareManager):response: Response,request: Request,spider: Spider,- ) -> Deferred[MutableChain[_T] | MutableAsyncChain[_T]]: # pragma: no cover+ ) -> Deferred[MutableAsyncChain[_T]]: # pragma: no coverwarn("SpiderMiddlewareManager.scrape_response() is deprecated, use scrape_response_async() instead",ScrapyDeprecationWarning,@@ -363,7 +211,7 @@ class SpiderMiddlewareManager(MiddlewareManager):scrape_func: ScrapeFunc[_T],response: Response,request: Request,- ) -> MutableChain[_T] | MutableAsyncChain[_T]:+ ) -> MutableAsyncChain[_T]:if not self.crawler:raise RuntimeError("scrape_response_async() called on a SpiderMiddlewareManager"@@ -373,7 +221,8 @@ class SpiderMiddlewareManager(MiddlewareManager):it: Iterable[_T] | AsyncIterator[_T] = await self._process_spider_input(scrape_func, response, request)- return await self._process_callback_output(response, it)+ ait = it if isinstance(it, AsyncIterator) else as_async_generator(it)+ return await self._process_callback_output(response, ait)except Exception as ex:await _defer_sleep_async()return self._process_spider_exception(response, ex)@@ -399,27 +248,23 @@ class SpiderMiddlewareManager(MiddlewareManager):# This method is only needed until _async compatibility methods are removed.@staticmethod- def _get_async_method_pair(- mw: Any, methodname: str- ) -> Callable | tuple[Callable, Callable] | None:- normal_method: Callable | None = getattr(mw, methodname, None)- methodname_async = methodname + "_async"- async_method: Callable | None = getattr(mw, methodname_async, None)+ def _get_process_spider_output(mw: Any) -> Callable | None:+ normal_method: Callable | None = getattr(mw, "process_spider_output", None)+ async_method: Callable | None = getattr(mw, "process_spider_output_async", None)if not async_method:if normal_method and not isasyncgenfunction(normal_method):- logger.warning(+ raise TypeError(f"Middleware {global_object_name(mw.__class__)} doesn't support"- f" asynchronous spider output, this is deprecated and will stop"- f" working in a future version of Scrapy. The middleware should"- f" be updated to support it. Please see"- f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"- f" for more information."+ f" asynchronous spider output. Its process_spider_output() method"+ f" should be an async generator function or it should additionally"+ f" define a process_spider_output_async() method.")return normal_methodif not normal_method:logger.error(- f"Middleware {global_object_name(mw.__class__)} has {methodname_async} "- f"without {methodname}, skipping this method."+ f"Middleware {global_object_name(mw.__class__)} has"+ f" process_spider_output_async() without process_spider_output(),"+ f" skipping this method. Please rename it to process_spider_output().")return Noneif not isasyncgenfunction(async_method):@@ -431,8 +276,8 @@ class SpiderMiddlewareManager(MiddlewareManager):if isasyncgenfunction(normal_method):logger.error(f"{global_object_name(normal_method)} is an async "- f"generator function while {methodname_async} exists, "- f"skipping both methods."+ f"generator function while process_spider_output_async() exists, "+ f"skipping both methods. Please remove process_spider_output_async().")return None- return normal_method, async_method+ return async_methoddiff --git a/scrapy/middleware.py b/scrapy/middleware.pyindex c4e0ce16e..14b591a5f 100644--- a/scrapy/middleware.py+++ b/scrapy/middleware.py@@ -50,10 +50,7 @@ class MiddlewareManager(ABC):)self.middlewares: tuple[Any, ...] = middlewares# Only process_spider_output and process_spider_exception can be None.- # Only process_spider_output can be a tuple, and only until _async compatibility methods are removed.- self.methods: dict[str, deque[Callable | tuple[Callable, Callable] | None]] = (- defaultdict(deque)- )+ self.methods: dict[str, deque[Callable | None]] = defaultdict(deque)self._mw_methods_requiring_spider: set[Callable] = set()for mw in middlewares:self._add_middleware(mw)diff --git a/scrapy/utils/python.py b/scrapy/utils/python.pyindex 9cb695111..2d40e8555 100644--- a/scrapy/utils/python.py+++ b/scrapy/utils/python.py@@ -8,12 +8,14 @@ import gcimport inspectimport reimport sys+import warningsimport weakreffrom collections.abc import AsyncIterator, Iterable, Mappingfrom functools import partial, wrapsfrom itertools import chainfrom typing import TYPE_CHECKING, Any, Concatenate, ParamSpec, TypeVar, overload+from scrapy.exceptions import ScrapyDeprecationWarningfrom scrapy.utils.asyncgen import as_async_generatorif TYPE_CHECKING:@@ -294,12 +296,13 @@ else:gc.collect()-class MutableChain(Iterable[_T]):- """- Thin wrapper around itertools.chain, allowing to add iterables "in-place"- """-+class MutableChain(Iterable[_T]): # pragma: no coverdef __init__(self, *args: Iterable[_T]):+ warnings.warn(+ "MutableChain is deprecated and will be removed in a future Scrapy version.",+ category=ScrapyDeprecationWarning,+ stacklevel=2,+ )self.data: Iterator[_T] = chain.from_iterable(args)def extend(self, *iterables: Iterable[_T]) -> None:diff --git a/tests/test_request_cb_kwargs.py b/tests/test_request_cb_kwargs.pyindex 8d3977452..0d96e1d88 100644--- a/tests/test_request_cb_kwargs.py+++ b/tests/test_request_cb_kwargs.py@@ -38,8 +38,8 @@ class InjectArgumentsSpiderMiddleware:if request.callback.__name__ == "parse_spider_mw":request.cb_kwargs["from_process_spider_input"] = True- def process_spider_output(self, response, result):- for element in result:+ async def process_spider_output(self, response, result):+ async for element in result:if (isinstance(element, Request)and element.callback.__name__ == "parse_spider_mw_2"diff --git a/tests/test_spidermiddleware.py b/tests/test_spidermiddleware.pyindex bfa60ac9c..2e0436ae1 100644--- a/tests/test_spidermiddleware.py+++ b/tests/test_spidermiddleware.py@@ -6,7 +6,6 @@ from typing import TYPE_CHECKING, Any, castfrom unittest import mockimport pytest-from testfixtures import LogCapturefrom twisted.internet import deferfrom scrapy.core.spidermw import SpiderMiddlewareManager@@ -63,20 +62,6 @@ class TestProcessSpiderInputInvalidOutput(TestSpiderMiddleware):await self._scrape_response()-class TestProcessSpiderOutputInvalidOutput(TestSpiderMiddleware):- """Invalid return value for process_spider_output method"""-- @coroutine_test- async def test_invalid_process_spider_output(self):- class InvalidProcessSpiderOutputMiddleware:- def process_spider_output(self, response, result):- return 1-- self.mwman._add_middleware(InvalidProcessSpiderOutputMiddleware())- with pytest.raises(_InvalidOutput):- await self._scrape_response()--class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):"""Invalid return value for process_spider_exception method"""@@ -87,13 +72,15 @@ class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):return 1class RaiseExceptionProcessSpiderOutputMiddleware:- def process_spider_output(self, response, result):+ async def process_spider_output(self, response, result):raise RuntimeError+ yield # pylint: disable=unreachableself.mwman._add_middleware(InvalidProcessSpiderOutputExceptionMiddleware())self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())+ it = await self._scrape_response()with pytest.raises(_InvalidOutput):- await self._scrape_response()+ await collect_asyncgen(it)class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):@@ -106,13 +93,15 @@ class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):return Noneclass RaiseExceptionProcessSpiderOutputMiddleware:- def process_spider_output(self, response, result):+ async def process_spider_output(self, response, result):1 / 0+ yieldself.mwman._add_middleware(ProcessSpiderExceptionReturnNoneMiddleware())self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())+ it = await self._scrape_response()with pytest.raises(ZeroDivisionError):- await self._scrape_response()+ await collect_asyncgen(it)class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):@@ -155,43 +144,19 @@ class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):self._scrape_func, self.response, self.request)- async def _test_simple_base(- self,- *mw_classes: type[Any],- downgrade: bool = False,- start_index: int | None = None,- ) -> None:- with LogCapture() as log:- result = await self._get_middleware_result(- *mw_classes, start_index=start_index- )- assert isinstance(result, Iterable)- result_list = list(result)- assert len(result_list) == self.RESULT_COUNT- assert isinstance(result_list[0], self.ITEM_TYPE)- assert ("downgraded to a non-async" in str(log)) == downgrade- assert ("doesn't support asynchronous spider output" in str(log)) == (- ProcessSpiderOutputSimpleMiddleware in mw_classes- )-async def _test_asyncgen_base(self,*mw_classes: type[Any],- downgrade: bool = False,start_index: int | None = None,) -> None:- with LogCapture() as log:- result = await self._get_middleware_result(- *mw_classes, start_index=start_index- )+ result = await self._get_middleware_result(*mw_classes, start_index=start_index)assert isinstance(result, AsyncIterator)result_list = await collect_asyncgen(result)assert len(result_list) == self.RESULT_COUNTassert isinstance(result_list[0], self.ITEM_TYPE)- assert ("downgraded to a non-async" in str(log)) == downgrade-class ProcessSpiderOutputSimpleMiddleware:+class ProcessSpiderOutputSyncMiddleware:def process_spider_output(self, response, result):yield from result@@ -232,53 +197,36 @@ class TestProcessSpiderOutputSimple(TestBaseAsyncSpiderMiddleware):"""process_spider_output tests for simple callbacks"""ITEM_TYPE = dict- MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware+ MW_SYNC = ProcessSpiderOutputSyncMiddlewareMW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddlewareMW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware@coroutine_test- async def test_simple(self):- """Simple mw"""- await self._test_simple_base(self.MW_SIMPLE)+ async def test_sync(self):+ """Unsupported sync mw"""+ with pytest.raises(+ TypeError, match=r"doesn't support asynchronous spider output"+ ):+ await self._get_middleware_result(self.MW_SYNC)@coroutine_testasync def test_asyncgen(self):- """Asyncgen mw; upgrade"""+ """Asyncgen mw"""await self._test_asyncgen_base(self.MW_ASYNCGEN)- @coroutine_test- async def test_simple_asyncgen(self):- """Simple mw -> asyncgen mw; upgrade"""- await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE)-- @coroutine_test- async def test_asyncgen_simple(self):- """Asyncgen mw -> simple mw; upgrade then downgrade"""- await self._test_simple_base(self.MW_SIMPLE, self.MW_ASYNCGEN, downgrade=True)-@coroutine_testasync def test_universal(self):"""Universal mw"""- await self._test_simple_base(self.MW_UNIVERSAL)-- @coroutine_test- async def test_universal_simple(self):- """Universal mw -> simple mw"""- await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL)-- @coroutine_test- async def test_simple_universal(self):- """Simple mw -> universal mw"""- await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE)+ await self._test_asyncgen_base(self.MW_UNIVERSAL)@coroutine_testasync def test_universal_asyncgen(self):- """Universal mw -> asyncgen mw; upgrade"""+ """Universal mw -> asyncgen mw"""await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_UNIVERSAL)@coroutine_testasync def test_asyncgen_universal(self):- """Asyncgen mw -> universal mw; upgrade"""+ """Asyncgen mw -> universal mw"""await self._test_asyncgen_base(self.MW_UNIVERSAL, self.MW_ASYNCGEN)@@ -289,31 +237,6 @@ class TestProcessSpiderOutputAsyncGen(TestProcessSpiderOutputSimple):for item in super()._callback():yield item- @coroutine_test- async def test_simple(self):- """Simple mw; downgrade"""- await self._test_simple_base(self.MW_SIMPLE, downgrade=True)-- @coroutine_test- async def test_simple_asyncgen(self):- """Simple mw -> asyncgen mw; downgrade then upgrade"""- await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE, downgrade=True)-- @coroutine_test- async def test_universal(self):- """Universal mw"""- await self._test_asyncgen_base(self.MW_UNIVERSAL)-- @coroutine_test- async def test_universal_simple(self):- """Universal mw -> simple mw; downgrade"""- await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL, downgrade=True)-- @coroutine_test- async def test_simple_universal(self):- """Simple mw -> universal mw; downgrade"""- await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE, downgrade=True)-class ProcessSpiderOutputNonIterableMiddleware:def process_spider_output(self, response, result):@@ -325,24 +248,6 @@ class ProcessSpiderOutputCoroutineMiddleware:return result-class TestProcessSpiderOutputInvalidResult(TestBaseAsyncSpiderMiddleware):- @coroutine_test- async def test_non_iterable(self):- with pytest.raises(- _InvalidOutput,- match=r"\.process_spider_output must return an iterable, got <class 'NoneType'>",- ):- await self._get_middleware_result(ProcessSpiderOutputNonIterableMiddleware)-- @coroutine_test- async def test_coroutine(self):- with pytest.raises(- _InvalidOutput,- match=r"\.process_spider_output must be an asynchronous generator",- ):- await self._get_middleware_result(ProcessSpiderOutputCoroutineMiddleware)--class ProcessStartSimpleMiddleware:async def process_start(self, start):async for item_or_request in start:@@ -415,11 +320,11 @@ class TestUniversalMiddlewareManager:return SpiderMiddlewareManager.from_crawler(crawler)def test_simple_mw(self, mwman: SpiderMiddlewareManager) -> None:- mw = ProcessSpiderOutputSimpleMiddleware()- mwman._add_middleware(mw)- assert (- mwman.methods["process_spider_output"][0] == mw.process_spider_output # pylint: disable=comparison-with-callable- )+ mw = ProcessSpiderOutputSyncMiddleware()+ with pytest.raises(+ TypeError, match=r"doesn't support asynchronous spider output"+ ):+ mwman._add_middleware(mw)def test_async_mw(self, mwman: SpiderMiddlewareManager) -> None:mw = ProcessSpiderOutputAsyncGenMiddleware()@@ -431,9 +336,8 @@ class TestUniversalMiddlewareManager:def test_universal_mw(self, mwman: SpiderMiddlewareManager) -> None:mw = ProcessSpiderOutputUniversalMiddleware()mwman._add_middleware(mw)- assert mwman.methods["process_spider_output"][0] == (- mw.process_spider_output,- mw.process_spider_output_async,+ assert (+ mwman.methods["process_spider_output"][0] == mw.process_spider_output_async # pylint: disable=comparison-with-callable)def test_universal_mw_no_sync(@@ -441,7 +345,7 @@ class TestUniversalMiddlewareManager:) -> None:mwman._add_middleware(UniversalMiddlewareNoSync())assert (- "UniversalMiddlewareNoSync has process_spider_output_async"+ "UniversalMiddlewareNoSync has process_spider_output_async()"" without process_spider_output" in caplog.text)assert mwman.methods["process_spider_output"][0] is None@@ -465,7 +369,7 @@ class TestUniversalMiddlewareManager:mwman._add_middleware(UniversalMiddlewareBothAsync())assert ("UniversalMiddlewareBothAsync.process_spider_output "- "is an async generator function while process_spider_output_async exists"+ "is an async generator function while process_spider_output_async() exists"in caplog.text)assert mwman.methods["process_spider_output"][0] is None@@ -473,7 +377,6 @@ class TestUniversalMiddlewareManager:class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):ITEM_TYPE = dict- MW_SIMPLE = ProcessSpiderOutputSimpleMiddlewareMW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddlewareMW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware@@ -490,11 +393,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):@coroutine_testasync def test_just_builtin(self):- await self._test_simple_base()-- @coroutine_test- async def test_builtin_simple(self):- await self._test_simple_base(self.MW_SIMPLE, start_index=1000)+ await self._test_asyncgen_base()@coroutine_testasync def test_builtin_async(self):@@ -503,11 +402,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):@coroutine_testasync def test_builtin_universal(self):- await self._test_simple_base(self.MW_UNIVERSAL, start_index=1000)-- @coroutine_test- async def test_simple_builtin(self):- await self._test_simple_base(self.MW_SIMPLE)+ await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)@coroutine_testasync def test_async_builtin(self):@@ -516,7 +411,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):@coroutine_testasync def test_universal_builtin(self):- await self._test_simple_base(self.MW_UNIVERSAL)+ await self._test_asyncgen_base(self.MW_UNIVERSAL)class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):@@ -524,40 +419,9 @@ class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):for item in super()._callback():yield item- @coroutine_test- async def test_just_builtin(self):- await self._test_asyncgen_base()-- @coroutine_test- async def test_builtin_simple(self):- """Downgrade"""- await self._test_simple_base(self.MW_SIMPLE, downgrade=True, start_index=1000)-- @coroutine_test- async def test_builtin_async(self):- await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)-- @coroutine_test- async def test_builtin_universal(self):- await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)-- @coroutine_test- async def test_simple_builtin(self):- """Downgrade"""- await self._test_simple_base(self.MW_SIMPLE, downgrade=True)-- @coroutine_test- async def test_async_builtin(self):- await self._test_asyncgen_base(self.MW_ASYNCGEN)-- @coroutine_test- async def test_universal_builtin(self):- await self._test_asyncgen_base(self.MW_UNIVERSAL)-class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):ITEM_TYPE = dict- MW_SIMPLE = ProcessSpiderOutputSimpleMiddlewareMW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddlewareMW_UNIVERSAL = ProcessSpiderOutputUniversalMiddlewareMW_EXC_SIMPLE = ProcessSpiderExceptionSimpleIterableMiddleware@@ -576,18 +440,13 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):@coroutine_testasync def test_exc_simple(self):"""Simple exc mw"""- await self._test_simple_base(self.MW_EXC_SIMPLE)+ await self._test_asyncgen_base(self.MW_EXC_SIMPLE)@coroutine_testasync def test_exc_async(self):"""Async exc mw"""await self._test_asyncgen_base(self.MW_EXC_ASYNCGEN)- @coroutine_test- async def test_exc_simple_simple(self):- """Simple exc mw -> simple output mw"""- await self._test_simple_base(self.MW_SIMPLE, self.MW_EXC_SIMPLE)-@coroutine_testasync def test_exc_async_async(self):"""Async exc mw -> async output mw"""@@ -598,25 +457,27 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):"""Simple exc mw -> async output mw; upgrade"""await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_EXC_SIMPLE)- @coroutine_test- async def test_exc_async_simple(self):- """Async exc mw -> simple output mw; cannot work as downgrading is not supported"""- await self._test_asyncgen_nodowngrade(self.MW_SIMPLE, self.MW_EXC_ASYNCGEN)-class TestDeprecatedSpiderArg(TestSpiderMiddleware):@coroutine_testasync def test_deprecated_mw_spider_arg(self):- class DeprecatedSpiderArgMiddleware:+ class DeprecatedSpiderArgMiddleware1:def process_spider_input(self, response, spider):return None- def process_spider_output(self, response, result, spider):+ async def process_spider_output(self, response, result, spider):1 / 0+ yield+ class DeprecatedSpiderArgMiddleware2:def process_spider_exception(self, response, exception, spider):return []+ with pytest.warns(+ ScrapyDeprecationWarning,+ match=r"process_spider_exception\(\) requires a spider argument",+ ):+ self.mwman._add_middleware(DeprecatedSpiderArgMiddleware2())with (pytest.warns(ScrapyDeprecationWarning,@@ -626,13 +487,10 @@ class TestDeprecatedSpiderArg(TestSpiderMiddleware):ScrapyDeprecationWarning,match=r"process_spider_output\(\) requires a spider argument",),- pytest.warns(- ScrapyDeprecationWarning,- match=r"process_spider_exception\(\) requires a spider argument",- ),):- self.mwman._add_middleware(DeprecatedSpiderArgMiddleware())- await self._scrape_response()+ self.mwman._add_middleware(DeprecatedSpiderArgMiddleware1())+ it = await self._scrape_response()+ await collect_asyncgen(it)@coroutine_testasync def test_deprecated_mwman_spider_arg(self):diff --git a/tests/test_spidermiddleware_output_chain.py b/tests/test_spidermiddleware_output_chain.pyindex 8cf08dd94..afdfaa322 100644--- a/tests/test_spidermiddleware_output_chain.py+++ b/tests/test_spidermiddleware_output_chain.py@@ -169,8 +169,8 @@ class NotGeneratorCallbackSpiderMiddlewareRightAfterSpider(NotGeneratorCallbackS# ================================================================================# (4) exceptions from a middleware process_spider_output method (generator)class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- for r in result:+ async def process_spider_output(self, response, result):+ async for r in result:r["processed"].append(f"{self.__class__.__name__}.process_spider_output")yield r@@ -182,8 +182,8 @@ class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):class GeneratorFailMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- for r in result:+ async def process_spider_output(self, response, result):+ async for r in result:r["processed"].append(f"{self.__class__.__name__}.process_spider_output")yield rraise LookupError@@ -201,8 +201,8 @@ class GeneratorDoNothingAfterFailureMiddleware(_GeneratorDoNothingMiddleware):class GeneratorRecoverMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- for r in result:+ async def process_spider_output(self, response, result):+ async for r in result:r["processed"].append(f"{self.__class__.__name__}.process_spider_output")yield r@@ -237,86 +237,6 @@ class GeneratorOutputChainSpider(Spider):yield {"processed": ["parse-second-item"]}-# ================================================================================-# (5) exceptions from a middleware process_spider_output method (not generator)---class _NotGeneratorDoNothingMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- out = []- for r in result:- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")- out.append(r)- return out-- def process_spider_exception(self, response, exception):- method = f"{self.__class__.__name__}.process_spider_exception"- self.crawler.spider.logger.info(- "%s: %s caught", method, exception.__class__.__name__- )---class NotGeneratorFailMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- out = []- for r in result:- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")- out.append(r)- raise ReferenceError-- def process_spider_exception(self, response, exception):- method = f"{self.__class__.__name__}.process_spider_exception"- self.crawler.spider.logger.info(- "%s: %s caught", method, exception.__class__.__name__- )- return [{"processed": [method]}]---class NotGeneratorDoNothingAfterFailureMiddleware(_NotGeneratorDoNothingMiddleware):- pass---class NotGeneratorRecoverMiddleware(_BaseSpiderMiddleware):- def process_spider_output(self, response, result):- out = []- for r in result:- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")- out.append(r)- return out-- def process_spider_exception(self, response, exception):- method = f"{self.__class__.__name__}.process_spider_exception"- self.crawler.spider.logger.info(- "%s: %s caught", method, exception.__class__.__name__- )- return [{"processed": [method]}]---class NotGeneratorDoNothingAfterRecoveryMiddleware(_NotGeneratorDoNothingMiddleware):- pass---class NotGeneratorOutputChainSpider(Spider):- name = "NotGeneratorOutputChainSpider"- custom_settings = {- "SPIDER_MIDDLEWARES": {- NotGeneratorFailMiddleware: 10,- NotGeneratorDoNothingAfterFailureMiddleware: 8,- NotGeneratorRecoverMiddleware: 5,- NotGeneratorDoNothingAfterRecoveryMiddleware: 3,- },- }-- async def start(self):- yield Request(self.mockserver.url("/status?n=200"))-- def parse(self, response):- return [- {"processed": ["parse-first-item"]},- {"processed": ["parse-second-item"]},- ]--# ================================================================================class TestSpiderMiddleware:mockserver: MockServer@@ -481,41 +401,3 @@ class TestSpiderMiddleware:assert str(item_from_callback) in str(log4)assert str(item_recovered) in str(log4)assert "parse-second-item" not in str(log4)-- @coroutine_test- async def test_not_a_generator_output_chain(self):- """- (5) An exception from a middleware's process_spider_output method should be sent- to the process_spider_exception method from the next middleware in the chain.- The result of the recovery by the process_spider_exception method should be handled- by the process_spider_output method from the next middleware.- The final item count should be 1 (from the process_spider_exception chain, the items- from the spider callback are lost)- """- log5 = await self.crawl_log(NotGeneratorOutputChainSpider)- assert "'item_scraped_count': 1" in str(log5)- assert (- "GeneratorRecoverMiddleware.process_spider_exception: ReferenceError caught"- in str(log5)- )- assert (- "GeneratorDoNothingAfterFailureMiddleware.process_spider_exception: ReferenceError caught"- in str(log5)- )- assert (- "GeneratorFailMiddleware.process_spider_exception: ReferenceError caught"- not in str(log5)- )- assert (- "GeneratorDoNothingAfterRecoveryMiddleware.process_spider_exception: ReferenceError caught"- not in str(log5)- )- item_recovered = {- "processed": [- "NotGeneratorRecoverMiddleware.process_spider_exception",- "NotGeneratorDoNothingAfterRecoveryMiddleware.process_spider_output",- ]- }- assert str(item_recovered) in str(log5)- assert "parse-first-item" not in str(log5)- assert "parse-second-item" not in str(log5)diff --git a/tests/test_utils_python.py b/tests/test_utils_python.pyindex 8dfac5f02..e8fe45749 100644--- a/tests/test_utils_python.py+++ b/tests/test_utils_python.py@@ -12,7 +12,6 @@ from scrapy.utils.asyncgen import as_async_generator, collect_asyncgenfrom scrapy.utils.defer import aiter_errbackfrom scrapy.utils.python import (MutableAsyncChain,- MutableChain,binary_is_text,get_func_args,memoizemethod_noargs,@@ -30,16 +29,6 @@ _KT = TypeVar("_KT")_VT = TypeVar("_VT")-def test_mutablechain():- m = MutableChain(range(2), [2, 3], (4, 5))- m.extend(range(6, 7))- m.extend([7, 8])- m.extend([9, 10], (11, 12))- assert next(m) == 0- assert m.__next__() == 1- assert list(m) == list(range(2, 13))--class TestMutableAsyncChain:@staticmethodasync def g1():