Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Remove support for sync process_spider_output(). (#7504)

Andrey Rakhmatullina8a8f20d9c49
diff --git a/docs/news.rst b/docs/news.rst
index 5eef10d84..3173dfbc1 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -1380,8 +1380,8 @@ Highlights:
- Added the :reqmeta:`allow_offsite` request meta key
-- :ref:`Spider middlewares that don't support asynchronous spider output
- <sync-async-spider-middleware>` are deprecated
+- Spider middlewares that don't support asynchronous spider output are
+ deprecated
- Added a base class for :ref:`universal spider middlewares
<universal-spider-middleware>`
@@ -1519,13 +1519,11 @@ Deprecations
``start_queue_cls`` parameter.
(:issue:`6752`)
-- :ref:`Spider middlewares that don't support asynchronous spider output
- <sync-async-spider-middleware>` are deprecated. The async iterable
- downgrading feature, needed for using such middlewares with asynchronous
- callbacks and with other spider middlewares that produce asynchronous
- iterables, is also deprecated. Please update all such middlewares to
- support asynchronous spider output.
- (:issue:`6664`)
+- Spider middlewares that don't support asynchronous spider output are
+ deprecated. The async iterable downgrading feature, needed for using such
+ middlewares with asynchronous callbacks and with other spider middlewares
+ that produce asynchronous iterables, is also deprecated. Please update all
+ such middlewares to support asynchronous spider output. (:issue:`6664`)
- Functions that were imported from :mod:`w3lib.url` and re-exported in
:mod:`scrapy.utils.url` are now deprecated, you should import them from
@@ -1784,9 +1782,8 @@ Documentation
- Documented the setting values set in the default project template.
(:issue:`6762`, :issue:`6775`)
-- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous
- iterable support in spider middlewares.
- (:issue:`6688`)
+- Improved the docs about asynchronous iterable support in spider
+ middlewares. (:issue:`6688`)
- Improved the :ref:`docs <coroutine-deferred-apis>` about using
:class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index ea3a60b4c..ba68f0dbc 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -23,9 +23,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.Request` callbacks.
- If you are using any custom or third-party :ref:`spider middleware
- <topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
-
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@@ -39,13 +36,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- The
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
- method of :ref:`spider middlewares <topics-spider-middleware>`.
-
- If defined as a coroutine, it must be an :term:`asynchronous generator`.
- The input ``result`` parameter is an :term:`asynchronous iterable`.
-
- See also :ref:`sync-async-spider-middleware` and
- :ref:`universal-spider-middleware`.
+ method of :ref:`spider middlewares <topics-spider-middleware>`, which
+ *must* be defined as an :term:`asynchronous generator` except in
+ :ref:`universal spider middlewares <universal-spider-middleware>`.
- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method
of :ref:`spider middlewares <custom-spider-middleware>`, which *must* be
@@ -277,136 +270,3 @@ You can also send multiple requests in parallel:
"price": responses[0][1].css(".price::text").get(),
"price2": responses[1][1].css(".color::text").get(),
}
-
-
-.. _sync-async-spider-middleware:
-
-Mixing synchronous and asynchronous spider middlewares
-======================================================
-
-The output of a :class:`~scrapy.Request` callback is passed as the ``result``
-parameter to the
-:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
-of the first :ref:`spider middleware <topics-spider-middleware>` from the
-:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.
-Then the output of that ``process_spider_output`` method is passed to the
-``process_spider_output`` method of the next spider middleware, and so on for
-every active spider middleware.
-
-Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods
-in this chain of calls.
-
-However, if any of the ``process_spider_output`` methods is defined as a
-synchronous method, and the previous ``Request`` callback or
-``process_spider_output`` method is a coroutine, there are some drawbacks to
-the asynchronous-to-synchronous conversion that Scrapy does so that the
-synchronous ``process_spider_output`` method gets a synchronous iterable as its
-``result`` parameter:
-
-- The whole output of the previous ``Request`` callback or
- ``process_spider_output`` method is awaited at this point.
-
-- If an exception raises while awaiting the output of the previous
- ``Request`` callback or ``process_spider_output`` method, none of that
- output will be processed.
-
- This contrasts with the regular behavior, where all items yielded before
- an exception raises are processed.
-
-Asynchronous-to-synchronous conversions are supported for backward
-compatibility, but they are deprecated and will stop working in a future
-version of Scrapy.
-
-To avoid asynchronous-to-synchronous conversions, when defining ``Request``
-callbacks as coroutine methods or when using spider middlewares whose
-``process_spider_output`` method is an :term:`asynchronous generator`, all
-active spider middlewares must either have their ``process_spider_output``
-method defined as an asynchronous generator or :ref:`define a
-process_spider_output_async method <universal-spider-middleware>`.
-
-.. _sync-async-spider-middleware-users:
-
-For middleware users
---------------------
-
-If you have asynchronous callbacks or use asynchronous-only spider middlewares
-you should make sure the asynchronous-to-synchronous conversions
-:ref:`described above <sync-async-spider-middleware>` don't happen. To do this,
-make sure all spider middlewares you use support asynchronous spider output.
-Even if you don't have asynchronous callbacks and don't use asynchronous-only
-spider middlewares in your project, it's still a good idea to make sure all
-middlewares you use support asynchronous spider output, so that it will be easy
-to start using asynchronous callbacks in the future. Because of this, Scrapy
-logs a warning when it detects a synchronous-only spider middleware.
-
-If you want to update middlewares you wrote, see the :ref:`following section
-<sync-async-spider-middleware-authors>`. If you have 3rd-party middlewares that
-aren't yet updated by their authors, you can :ref:`subclass <tut-inheritance>`
-them to make them :ref:`universal <universal-spider-middleware>` and use the
-subclasses in your projects.
-
-.. _sync-async-spider-middleware-authors:
-
-For middleware authors
-----------------------
-
-If you have a spider middleware that defines a synchronous
-``process_spider_output`` method, you should update it to support asynchronous
-spider output for :ref:`better compatibility <sync-async-spider-middleware>`,
-even if you don't yet use it with asynchronous callbacks, especially if you
-publish this middleware for other people to use. You have two options for this:
-
-1. Make the middleware asynchronous, by making the ``process_spider_output``
- method an :term:`asynchronous generator`.
-2. Make the middleware universal, as described in the :ref:`next section
- <universal-spider-middleware>`.
-
-If your middleware won't be used in projects with synchronous-only middlewares,
-e.g. because it's an internal middleware and you know that all other
-middlewares in your projects are already updated, it's safe to choose the first
-option. Otherwise, it's better to choose the second option.
-
-.. _universal-spider-middleware:
-
-Universal spider middlewares
-----------------------------
-
-To allow writing a spider middleware that supports asynchronous execution of
-its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding
-:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
-while maintaining support for older Scrapy versions, you may define
-``process_spider_output`` as a synchronous method and define an
-:term:`asynchronous generator` version of that method with an alternative name:
-``process_spider_output_async``.
-
-For example:
-
-.. code-block:: python
-
- class UniversalSpiderMiddleware:
- def process_spider_output(self, response, result):
- for r in result:
- # ... do something with r
- yield r
-
- async def process_spider_output_async(self, response, result):
- async for r in result:
- # ... do something with r
- yield r
-
-.. note:: This is an interim measure to allow, for a time, to write code that
- works in Scrapy 2.7 and later without requiring
- asynchronous-to-synchronous conversions, and works in earlier Scrapy
- versions as well.
-
- In some future version of Scrapy, however, this feature will be
- deprecated and, eventually, in a later version of Scrapy, this
- feature will be removed, and all spider middlewares will be expected
- to define their ``process_spider_output`` method as an asynchronous
- generator.
-
-Since 2.13.0, Scrapy provides a base class,
-:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements
-the ``process_spider_output()`` and ``process_spider_output_async()`` methods,
-so instead of duplicating the processing code you can override the
-``get_processed_request()`` and/or the ``get_processed_item()`` method.
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 8b000697a..820d5910c 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -117,36 +117,28 @@ one or more of these methods:
:type response: :class:`~scrapy.http.Response` object
.. method:: process_spider_output(response, result)
+ :async:
- This method is called with the results returned from the Spider, after
- it has processed the response.
-
- :meth:`process_spider_output` must return an iterable of
- :class:`~scrapy.Request` objects and :ref:`item objects
- <topics-items>`.
+ This method is an :term:`asynchronous generator` called with the
+ results from the spider after the spider has processed the response.
- Consider defining this method as an :term:`asynchronous generator`,
- which will be a requirement in a future version of Scrapy. However, if
- you plan on sharing your spider middleware with other people, consider
- either :ref:`enforcing Scrapy 2.7 <enforce-component-requirements>`
- as a minimum requirement of your spider middleware, or :ref:`making
- your spider middleware universal <universal-spider-middleware>` so that
- it works with Scrapy versions earlier than Scrapy 2.7.
+ .. seealso:: :ref:`universal-spider-middleware`.
:param response: the response which generated this output from the
spider
:type response: :class:`~scrapy.http.Response` object
- :param result: the result returned by the spider
- :type result: an iterable of :class:`~scrapy.Request` objects and
- :ref:`item objects <topics-items>`
+ :param result: the results from the spider
+ :type result: an :term:`asynchronous iterable` of
+ :class:`~scrapy.Request` objects and :ref:`item objects
+ <topics-items>`
.. method:: process_spider_output_async(response, result)
:async:
- If defined, this method must be an :term:`asynchronous generator`,
- which will be called instead of :meth:`process_spider_output` if
- ``result`` is an :term:`asynchronous iterable`.
+ Alternative name for :meth:`process_spider_output` used when
+ implementing a :ref:`universal spider middleware
+ <universal-spider-middleware>`.
.. method:: process_spider_exception(response, exception)
@@ -174,13 +166,40 @@ one or more of these methods:
:type exception: :exc:`Exception` object
+.. _universal-spider-middleware:
+
+Universal spider middlewares
+----------------------------
+
+In Scrapy 2.6.3 and lower, ``process_spider_output()`` must be a *synchronous*
+generator.
+
+To support those versions and higher Scrapy versions in the same middleware,
+rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()`
+method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define
+a synchronous ``process_spider_output()`` method to be used by 2.6.3 and lower
+versions.
+
+For example:
+
+.. code-block:: python
+
+ class UniversalSpiderMiddleware:
+ async def process_spider_output_async(self, response, result):
+ async for r in result:
+ # ... do something with r
+ yield r
+
+ def process_spider_output(self, response, result):
+ for r in result:
+ # ... do something with r
+ yield r
+
Base class for custom spider middlewares
----------------------------------------
Scrapy provides a base class for custom spider middlewares. It's not required
-to use it but it can help with simplifying middleware implementations and
-reducing the amount of boilerplate code in :ref:`universal middlewares
-<universal-spider-middleware>`.
+to use it but it can help with simplifying middleware implementations.
.. module:: scrapy.spidermiddlewares.base
diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py
index d48a65967..bcf38bb75 100644
--- a/scrapy/core/spidermw.py
+++ b/scrapy/core/spidermw.py
@@ -9,29 +9,28 @@ from __future__ import annotations
import logging
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable
from functools import wraps
-from inspect import isasyncgenfunction, iscoroutine
+from inspect import isasyncgenfunction
from itertools import islice
from typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, cast
from warnings import warn
-from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.python.failure import Failure
from scrapy import Request, Spider
from scrapy.exceptions import ScrapyDeprecationWarning, _InvalidOutput
from scrapy.http import Response
from scrapy.middleware import MiddlewareManager
-from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
+from scrapy.utils.asyncgen import as_async_generator
from scrapy.utils.conf import build_component_list
from scrapy.utils.defer import (
_defer_sleep_async,
deferred_from_coro,
maybe_deferred_to_future,
)
-from scrapy.utils.python import MutableAsyncChain, MutableChain, global_object_name
+from scrapy.utils.python import MutableAsyncChain, global_object_name
if TYPE_CHECKING:
- from collections.abc import Generator
+ from twisted.internet.defer import Deferred
from scrapy.settings import BaseSettings
@@ -46,10 +45,6 @@ ScrapeFunc: TypeAlias = Callable[
]
-def _isiterable(o: Any) -> bool:
- return isinstance(o, (Iterable, AsyncIterator))
-
-
class SpiderMiddlewareManager(MiddlewareManager):
component_name = "spider middleware"
@@ -67,13 +62,10 @@ class SpiderMiddlewareManager(MiddlewareManager):
if hasattr(mw, "process_start"):
self.methods["process_start"].appendleft(mw.process_start)
- process_spider_output = self._get_async_method_pair(mw, "process_spider_output")
+ process_spider_output = self._get_process_spider_output(mw)
self.methods["process_spider_output"].appendleft(process_spider_output)
- if callable(process_spider_output):
+ if process_spider_output is not None:
self._check_mw_method_spider_arg(process_spider_output)
- elif isinstance(process_spider_output, tuple):
- for m in process_spider_output:
- self._check_mw_method_spider_arg(m)
process_spider_exception = getattr(mw, "process_spider_exception", None)
self.methods["process_spider_exception"].appendleft(process_spider_exception)
@@ -105,48 +97,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
return await scrape_func(Failure(), request)
return await scrape_func(response, request)
- def _evaluate_iterable(
- self,
- response: Response,
- iterable: Iterable[_T] | AsyncIterator[_T],
- exception_processor_index: int,
- recover_to: MutableChain[_T] | MutableAsyncChain[_T],
- ) -> Iterable[_T] | AsyncIterator[_T]:
-
- if isinstance(iterable, AsyncIterator):
- return self._process_async(
- response,
- iterable,
- exception_processor_index,
- cast("MutableAsyncChain[_T]", recover_to),
- )
- return self._process_sync(
- response,
- iterable,
- exception_processor_index,
- cast("MutableChain[_T]", recover_to),
- )
-
- def _process_sync(
- self,
- response: Response,
- iterable: Iterable[_T],
- exception_processor_index: int,
- recover_to: MutableChain[_T],
- ) -> Iterable[_T]:
- try:
- yield from iterable
- except Exception as ex:
- exception_result = cast(
- "Failure | MutableChain[_T]",
- self._process_spider_exception(response, ex, exception_processor_index),
- )
- if isinstance(exception_result, Failure):
- raise
- assert isinstance(recover_to, MutableChain)
- recover_to.extend(exception_result)
-
- async def _process_async(
+ async def _evaluate_iterable(
self,
response: Response,
iterable: AsyncIterator[_T],
@@ -157,13 +108,9 @@ class SpiderMiddlewareManager(MiddlewareManager):
async for r in iterable:
yield r
except Exception as ex:
- exception_result = cast(
- "Failure | MutableAsyncChain[_T]",
- self._process_spider_exception(response, ex, exception_processor_index),
+ exception_result: MutableAsyncChain[_T] = self._process_spider_exception(
+ response, ex, exception_processor_index
)
- if isinstance(exception_result, Failure):
- raise
- assert isinstance(recover_to, MutableAsyncChain)
recover_to.extend(exception_result)
def _process_spider_exception(
@@ -171,7 +118,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
response: Response,
exception: Exception,
start_index: int = 0,
- ) -> MutableChain[_T] | MutableAsyncChain[_T]:
+ ) -> MutableAsyncChain[_T]:
# don't handle _InvalidOutput exception
if isinstance(exception, _InvalidOutput):
raise exception
@@ -181,28 +128,18 @@ class SpiderMiddlewareManager(MiddlewareManager):
for method_index, method in enumerate(method_list, start=start_index):
if method is None:
continue
- method = cast("Callable", method)
if method in self._mw_methods_requiring_spider:
result = method(
response=response, exception=exception, spider=self._spider
)
else:
result = method(response=response, exception=exception)
- if _isiterable(result):
+ if isinstance(result, (Iterable, AsyncIterator)):
# stop exception handling by handing control over to the
# process_spider_output chain if an iterable has been returned
- dfd: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = (
- self._process_spider_output(response, result, method_index + 1)
- )
- # _process_spider_output() returns a Deferred only because of downgrading so this can be
- # simplified when downgrading is removed.
- if dfd.called:
- # the result is available immediately if _process_spider_output didn't do downgrading
- return cast("MutableChain[_T] | MutableAsyncChain[_T]", dfd.result)
- # we forbid waiting here because otherwise we would need to return a deferred from
- # _process_spider_exception too, which complicates the architecture
- msg = f"Async iterable returned from {global_object_name(method)} cannot be downgraded"
- raise _InvalidOutput(msg)
+ if isinstance(result, Iterable):
+ result = as_async_generator(result)
+ return self._process_spider_output(response, result, method_index + 1)
if result is None:
continue
msg = (
@@ -212,124 +149,35 @@ class SpiderMiddlewareManager(MiddlewareManager):
raise _InvalidOutput(msg)
raise exception
- # This method cannot be made async def, as _process_spider_exception relies on the Deferred result
- # being available immediately which doesn't work when it's a wrapped coroutine.
- # It also needs @inlineCallbacks only because of downgrading so it can be removed when downgrading is removed.
- @inlineCallbacks
- def _process_spider_output( # noqa: PLR0912
+ def _process_spider_output(
self,
response: Response,
- result: Iterable[_T] | AsyncIterator[_T],
+ result: AsyncIterator[_T],
start_index: int = 0,
- ) -> Generator[Deferred[Any], Any, MutableChain[_T] | MutableAsyncChain[_T]]:
+ ) -> MutableAsyncChain[_T]:
# items in this iterable do not need to go through the process_spider_output
# chain, they went through it already from the process_spider_exception method
- recovered: MutableChain[_T] | MutableAsyncChain[_T]
- last_result_is_async = isinstance(result, AsyncIterator)
- recovered = MutableAsyncChain() if last_result_is_async else MutableChain()
-
- # There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.
- # 1. def foo. Sync iterables are passed as is, async ones are downgraded.
- # 2. async def foo. Sync iterables are upgraded, async ones are passed as is.
- # 3. def foo + async def foo_async. Iterables are passed to the respective method.
- # Storing methods and method tuples in the same list is weird but we should be able to roll this back
- # when we drop this compatibility feature.
-
+ recovered: MutableAsyncChain[_T] = MutableAsyncChain()
method_list = islice(self.methods["process_spider_output"], start_index, None)
- for method_index, method_pair in enumerate(method_list, start=start_index):
- if method_pair is None:
+ for method_index, method in enumerate(method_list, start=start_index):
+ if method is None:
continue
- need_upgrade = need_downgrade = False
- if isinstance(method_pair, tuple):
- # This tuple handling is only needed until _async compatibility methods are removed.
- method_sync, method_async = method_pair
- method = method_async if last_result_is_async else method_sync
- else:
- method = method_pair
- if not last_result_is_async and isasyncgenfunction(method):
- need_upgrade = True
- elif last_result_is_async and not isasyncgenfunction(method):
- need_downgrade = True
- try:
- if need_upgrade:
- # Iterable -> AsyncIterator
- result = as_async_generator(result)
- elif need_downgrade:
- logger.warning(
- f"Async iterable passed to {global_object_name(method)} was"
- f" downgraded to a non-async one. This is deprecated and will"
- f" stop working in a future version of Scrapy. Please see"
- f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
- f" for more information."
- )
- assert isinstance(result, AsyncIterator)
- # AsyncIterator -> Iterable
- result = yield deferred_from_coro(collect_asyncgen(result))
- if isinstance(recovered, AsyncIterator):
- recovered_collected = yield deferred_from_coro(
- collect_asyncgen(recovered)
- )
- recovered = MutableChain(recovered_collected)
- # might fail directly if the output value is not a generator
- if method in self._mw_methods_requiring_spider:
- result = method(
- response=response, result=result, spider=self._spider
- )
- else:
- result = method(response=response, result=result)
- except Exception as ex:
- exception_result: Failure | MutableChain[_T] | MutableAsyncChain[_T] = (
- self._process_spider_exception(response, ex, method_index + 1)
- )
- if isinstance(exception_result, Failure):
- raise
- return exception_result
- if _isiterable(result):
- result = self._evaluate_iterable(
- response, result, method_index + 1, recovered
- )
+ if method in self._mw_methods_requiring_spider:
+ result = method(response=response, result=result, spider=self._spider)
else:
- if iscoroutine(result):
- result.close() # Silence warning about not awaiting
- msg = (
- f"{global_object_name(method)} must be an asynchronous "
- f"generator (i.e. use yield)"
- )
- else:
- msg = (
- f"{global_object_name(method)} must return an iterable, got "
- f"{type(result)}"
- )
- raise _InvalidOutput(msg)
- last_result_is_async = isinstance(result, AsyncIterator)
-
- if last_result_is_async:
- return MutableAsyncChain(result, recovered)
- return MutableChain(result, recovered) # type: ignore[arg-type]
+ result = method(response=response, result=result)
+ result = self._evaluate_iterable(
+ response, result, method_index + 1, recovered
+ )
+ return MutableAsyncChain(result, recovered)
async def _process_callback_output(
- self,
- response: Response,
- result: Iterable[_T] | AsyncIterator[_T],
- ) -> MutableChain[_T] | MutableAsyncChain[_T]:
- recovered: MutableChain[_T] | MutableAsyncChain[_T]
- if isinstance(result, AsyncIterator):
- recovered = MutableAsyncChain()
- else:
- recovered = MutableChain()
+ self, response: Response, result: AsyncIterator[_T]
+ ) -> MutableAsyncChain[_T]:
+ recovered: MutableAsyncChain[_T] = MutableAsyncChain()
result = self._evaluate_iterable(response, result, 0, recovered)
- result = await maybe_deferred_to_future(
- cast(
- "Deferred[Iterable[_T] | AsyncIterator[_T]]",
- self._process_spider_output(response, result),
- )
- )
- if isinstance(result, AsyncIterator):
- return MutableAsyncChain(result, recovered)
- if isinstance(recovered, AsyncIterator):
- recovered_collected = await collect_asyncgen(recovered)
- recovered = MutableChain(recovered_collected)
- return MutableChain(result, recovered)
+ result = self._process_spider_output(response, result)
+ return MutableAsyncChain(result, recovered)
def scrape_response(
self,
@@ -340,7 +188,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
response: Response,
request: Request,
spider: Spider,
- ) -> Deferred[MutableChain[_T] | MutableAsyncChain[_T]]: # pragma: no cover
+ ) -> Deferred[MutableAsyncChain[_T]]: # pragma: no cover
warn(
"SpiderMiddlewareManager.scrape_response() is deprecated, use scrape_response_async() instead",
ScrapyDeprecationWarning,
@@ -363,7 +211,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
scrape_func: ScrapeFunc[_T],
response: Response,
request: Request,
- ) -> MutableChain[_T] | MutableAsyncChain[_T]:
+ ) -> MutableAsyncChain[_T]:
if not self.crawler:
raise RuntimeError(
"scrape_response_async() called on a SpiderMiddlewareManager"
@@ -373,7 +221,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
it: Iterable[_T] | AsyncIterator[_T] = await self._process_spider_input(
scrape_func, response, request
)
- return await self._process_callback_output(response, it)
+ ait = it if isinstance(it, AsyncIterator) else as_async_generator(it)
+ return await self._process_callback_output(response, ait)
except Exception as ex:
await _defer_sleep_async()
return self._process_spider_exception(response, ex)
@@ -399,27 +248,23 @@ class SpiderMiddlewareManager(MiddlewareManager):
# This method is only needed until _async compatibility methods are removed.
@staticmethod
- def _get_async_method_pair(
- mw: Any, methodname: str
- ) -> Callable | tuple[Callable, Callable] | None:
- normal_method: Callable | None = getattr(mw, methodname, None)
- methodname_async = methodname + "_async"
- async_method: Callable | None = getattr(mw, methodname_async, None)
+ def _get_process_spider_output(mw: Any) -> Callable | None:
+ normal_method: Callable | None = getattr(mw, "process_spider_output", None)
+ async_method: Callable | None = getattr(mw, "process_spider_output_async", None)
if not async_method:
if normal_method and not isasyncgenfunction(normal_method):
- logger.warning(
+ raise TypeError(
f"Middleware {global_object_name(mw.__class__)} doesn't support"
- f" asynchronous spider output, this is deprecated and will stop"
- f" working in a future version of Scrapy. The middleware should"
- f" be updated to support it. Please see"
- f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
- f" for more information."
+ f" asynchronous spider output. Its process_spider_output() method"
+ f" should be an async generator function or it should additionally"
+ f" define a process_spider_output_async() method."
)
return normal_method
if not normal_method:
logger.error(
- f"Middleware {global_object_name(mw.__class__)} has {methodname_async} "
- f"without {methodname}, skipping this method."
+ f"Middleware {global_object_name(mw.__class__)} has"
+ f" process_spider_output_async() without process_spider_output(),"
+ f" skipping this method. Please rename it to process_spider_output()."
)
return None
if not isasyncgenfunction(async_method):
@@ -431,8 +276,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
if isasyncgenfunction(normal_method):
logger.error(
f"{global_object_name(normal_method)} is an async "
- f"generator function while {methodname_async} exists, "
- f"skipping both methods."
+ f"generator function while process_spider_output_async() exists, "
+ f"skipping both methods. Please remove process_spider_output_async()."
)
return None
- return normal_method, async_method
+ return async_method
diff --git a/scrapy/middleware.py b/scrapy/middleware.py
index c4e0ce16e..14b591a5f 100644
--- a/scrapy/middleware.py
+++ b/scrapy/middleware.py
@@ -50,10 +50,7 @@ class MiddlewareManager(ABC):
)
self.middlewares: tuple[Any, ...] = middlewares
# Only process_spider_output and process_spider_exception can be None.
- # Only process_spider_output can be a tuple, and only until _async compatibility methods are removed.
- self.methods: dict[str, deque[Callable | tuple[Callable, Callable] | None]] = (
- defaultdict(deque)
- )
+ self.methods: dict[str, deque[Callable | None]] = defaultdict(deque)
self._mw_methods_requiring_spider: set[Callable] = set()
for mw in middlewares:
self._add_middleware(mw)
diff --git a/scrapy/utils/python.py b/scrapy/utils/python.py
index 9cb695111..2d40e8555 100644
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@@ -8,12 +8,14 @@ import gc
import inspect
import re
import sys
+import warnings
import weakref
from collections.abc import AsyncIterator, Iterable, Mapping
from functools import partial, wraps
from itertools import chain
from typing import TYPE_CHECKING, Any, Concatenate, ParamSpec, TypeVar, overload
+from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.utils.asyncgen import as_async_generator
if TYPE_CHECKING:
@@ -294,12 +296,13 @@ else:
gc.collect()
-class MutableChain(Iterable[_T]):
- """
- Thin wrapper around itertools.chain, allowing to add iterables "in-place"
- """
-
+class MutableChain(Iterable[_T]): # pragma: no cover
def __init__(self, *args: Iterable[_T]):
+ warnings.warn(
+ "MutableChain is deprecated and will be removed in a future Scrapy version.",
+ category=ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
self.data: Iterator[_T] = chain.from_iterable(args)
def extend(self, *iterables: Iterable[_T]) -> None:
diff --git a/tests/test_request_cb_kwargs.py b/tests/test_request_cb_kwargs.py
index 8d3977452..0d96e1d88 100644
--- a/tests/test_request_cb_kwargs.py
+++ b/tests/test_request_cb_kwargs.py
@@ -38,8 +38,8 @@ class InjectArgumentsSpiderMiddleware:
if request.callback.__name__ == "parse_spider_mw":
request.cb_kwargs["from_process_spider_input"] = True
- def process_spider_output(self, response, result):
- for element in result:
+ async def process_spider_output(self, response, result):
+ async for element in result:
if (
isinstance(element, Request)
and element.callback.__name__ == "parse_spider_mw_2"
diff --git a/tests/test_spidermiddleware.py b/tests/test_spidermiddleware.py
index bfa60ac9c..2e0436ae1 100644
--- a/tests/test_spidermiddleware.py
+++ b/tests/test_spidermiddleware.py
@@ -6,7 +6,6 @@ from typing import TYPE_CHECKING, Any, cast
from unittest import mock
import pytest
-from testfixtures import LogCapture
from twisted.internet import defer
from scrapy.core.spidermw import SpiderMiddlewareManager
@@ -63,20 +62,6 @@ class TestProcessSpiderInputInvalidOutput(TestSpiderMiddleware):
await self._scrape_response()
-class TestProcessSpiderOutputInvalidOutput(TestSpiderMiddleware):
- """Invalid return value for process_spider_output method"""
-
- @coroutine_test
- async def test_invalid_process_spider_output(self):
- class InvalidProcessSpiderOutputMiddleware:
- def process_spider_output(self, response, result):
- return 1
-
- self.mwman._add_middleware(InvalidProcessSpiderOutputMiddleware())
- with pytest.raises(_InvalidOutput):
- await self._scrape_response()
-
-
class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
"""Invalid return value for process_spider_exception method"""
@@ -87,13 +72,15 @@ class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
return 1
class RaiseExceptionProcessSpiderOutputMiddleware:
- def process_spider_output(self, response, result):
+ async def process_spider_output(self, response, result):
raise RuntimeError
+ yield # pylint: disable=unreachable
self.mwman._add_middleware(InvalidProcessSpiderOutputExceptionMiddleware())
self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
+ it = await self._scrape_response()
with pytest.raises(_InvalidOutput):
- await self._scrape_response()
+ await collect_asyncgen(it)
class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
@@ -106,13 +93,15 @@ class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
return None
class RaiseExceptionProcessSpiderOutputMiddleware:
- def process_spider_output(self, response, result):
+ async def process_spider_output(self, response, result):
1 / 0
+ yield
self.mwman._add_middleware(ProcessSpiderExceptionReturnNoneMiddleware())
self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
+ it = await self._scrape_response()
with pytest.raises(ZeroDivisionError):
- await self._scrape_response()
+ await collect_asyncgen(it)
class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
@@ -155,43 +144,19 @@ class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
self._scrape_func, self.response, self.request
)
- async def _test_simple_base(
- self,
- *mw_classes: type[Any],
- downgrade: bool = False,
- start_index: int | None = None,
- ) -> None:
- with LogCapture() as log:
- result = await self._get_middleware_result(
- *mw_classes, start_index=start_index
- )
- assert isinstance(result, Iterable)
- result_list = list(result)
- assert len(result_list) == self.RESULT_COUNT
- assert isinstance(result_list[0], self.ITEM_TYPE)
- assert ("downgraded to a non-async" in str(log)) == downgrade
- assert ("doesn't support asynchronous spider output" in str(log)) == (
- ProcessSpiderOutputSimpleMiddleware in mw_classes
- )
-
async def _test_asyncgen_base(
self,
*mw_classes: type[Any],
- downgrade: bool = False,
start_index: int | None = None,
) -> None:
- with LogCapture() as log:
- result = await self._get_middleware_result(
- *mw_classes, start_index=start_index
- )
+ result = await self._get_middleware_result(*mw_classes, start_index=start_index)
assert isinstance(result, AsyncIterator)
result_list = await collect_asyncgen(result)
assert len(result_list) == self.RESULT_COUNT
assert isinstance(result_list[0], self.ITEM_TYPE)
- assert ("downgraded to a non-async" in str(log)) == downgrade
-class ProcessSpiderOutputSimpleMiddleware:
+class ProcessSpiderOutputSyncMiddleware:
def process_spider_output(self, response, result):
yield from result
@@ -232,53 +197,36 @@ class TestProcessSpiderOutputSimple(TestBaseAsyncSpiderMiddleware):
"""process_spider_output tests for simple callbacks"""
ITEM_TYPE = dict
- MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
+ MW_SYNC = ProcessSpiderOutputSyncMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
@coroutine_test
- async def test_simple(self):
- """Simple mw"""
- await self._test_simple_base(self.MW_SIMPLE)
+ async def test_sync(self):
+ """Unsupported sync mw"""
+ with pytest.raises(
+ TypeError, match=r"doesn't support asynchronous spider output"
+ ):
+ await self._get_middleware_result(self.MW_SYNC)
@coroutine_test
async def test_asyncgen(self):
- """Asyncgen mw; upgrade"""
+ """Asyncgen mw"""
await self._test_asyncgen_base(self.MW_ASYNCGEN)
- @coroutine_test
- async def test_simple_asyncgen(self):
- """Simple mw -> asyncgen mw; upgrade"""
- await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE)
-
- @coroutine_test
- async def test_asyncgen_simple(self):
- """Asyncgen mw -> simple mw; upgrade then downgrade"""
- await self._test_simple_base(self.MW_SIMPLE, self.MW_ASYNCGEN, downgrade=True)
-
@coroutine_test
async def test_universal(self):
"""Universal mw"""
- await self._test_simple_base(self.MW_UNIVERSAL)
-
- @coroutine_test
- async def test_universal_simple(self):
- """Universal mw -> simple mw"""
- await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL)
-
- @coroutine_test
- async def test_simple_universal(self):
- """Simple mw -> universal mw"""
- await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE)
+ await self._test_asyncgen_base(self.MW_UNIVERSAL)
@coroutine_test
async def test_universal_asyncgen(self):
- """Universal mw -> asyncgen mw; upgrade"""
+ """Universal mw -> asyncgen mw"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_UNIVERSAL)
@coroutine_test
async def test_asyncgen_universal(self):
- """Asyncgen mw -> universal mw; upgrade"""
+ """Asyncgen mw -> universal mw"""
await self._test_asyncgen_base(self.MW_UNIVERSAL, self.MW_ASYNCGEN)
@@ -289,31 +237,6 @@ class TestProcessSpiderOutputAsyncGen(TestProcessSpiderOutputSimple):
for item in super()._callback():
yield item
- @coroutine_test
- async def test_simple(self):
- """Simple mw; downgrade"""
- await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
-
- @coroutine_test
- async def test_simple_asyncgen(self):
- """Simple mw -> asyncgen mw; downgrade then upgrade"""
- await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE, downgrade=True)
-
- @coroutine_test
- async def test_universal(self):
- """Universal mw"""
- await self._test_asyncgen_base(self.MW_UNIVERSAL)
-
- @coroutine_test
- async def test_universal_simple(self):
- """Universal mw -> simple mw; downgrade"""
- await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL, downgrade=True)
-
- @coroutine_test
- async def test_simple_universal(self):
- """Simple mw -> universal mw; downgrade"""
- await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE, downgrade=True)
-
class ProcessSpiderOutputNonIterableMiddleware:
def process_spider_output(self, response, result):
@@ -325,24 +248,6 @@ class ProcessSpiderOutputCoroutineMiddleware:
return result
-class TestProcessSpiderOutputInvalidResult(TestBaseAsyncSpiderMiddleware):
- @coroutine_test
- async def test_non_iterable(self):
- with pytest.raises(
- _InvalidOutput,
- match=r"\.process_spider_output must return an iterable, got <class 'NoneType'>",
- ):
- await self._get_middleware_result(ProcessSpiderOutputNonIterableMiddleware)
-
- @coroutine_test
- async def test_coroutine(self):
- with pytest.raises(
- _InvalidOutput,
- match=r"\.process_spider_output must be an asynchronous generator",
- ):
- await self._get_middleware_result(ProcessSpiderOutputCoroutineMiddleware)
-
-
class ProcessStartSimpleMiddleware:
async def process_start(self, start):
async for item_or_request in start:
@@ -415,11 +320,11 @@ class TestUniversalMiddlewareManager:
return SpiderMiddlewareManager.from_crawler(crawler)
def test_simple_mw(self, mwman: SpiderMiddlewareManager) -> None:
- mw = ProcessSpiderOutputSimpleMiddleware()
- mwman._add_middleware(mw)
- assert (
- mwman.methods["process_spider_output"][0] == mw.process_spider_output # pylint: disable=comparison-with-callable
- )
+ mw = ProcessSpiderOutputSyncMiddleware()
+ with pytest.raises(
+ TypeError, match=r"doesn't support asynchronous spider output"
+ ):
+ mwman._add_middleware(mw)
def test_async_mw(self, mwman: SpiderMiddlewareManager) -> None:
mw = ProcessSpiderOutputAsyncGenMiddleware()
@@ -431,9 +336,8 @@ class TestUniversalMiddlewareManager:
def test_universal_mw(self, mwman: SpiderMiddlewareManager) -> None:
mw = ProcessSpiderOutputUniversalMiddleware()
mwman._add_middleware(mw)
- assert mwman.methods["process_spider_output"][0] == (
- mw.process_spider_output,
- mw.process_spider_output_async,
+ assert (
+ mwman.methods["process_spider_output"][0] == mw.process_spider_output_async # pylint: disable=comparison-with-callable
)
def test_universal_mw_no_sync(
@@ -441,7 +345,7 @@ class TestUniversalMiddlewareManager:
) -> None:
mwman._add_middleware(UniversalMiddlewareNoSync())
assert (
- "UniversalMiddlewareNoSync has process_spider_output_async"
+ "UniversalMiddlewareNoSync has process_spider_output_async()"
" without process_spider_output" in caplog.text
)
assert mwman.methods["process_spider_output"][0] is None
@@ -465,7 +369,7 @@ class TestUniversalMiddlewareManager:
mwman._add_middleware(UniversalMiddlewareBothAsync())
assert (
"UniversalMiddlewareBothAsync.process_spider_output "
- "is an async generator function while process_spider_output_async exists"
+ "is an async generator function while process_spider_output_async() exists"
in caplog.text
)
assert mwman.methods["process_spider_output"][0] is None
@@ -473,7 +377,6 @@ class TestUniversalMiddlewareManager:
class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
ITEM_TYPE = dict
- MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
@@ -490,11 +393,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_just_builtin(self):
- await self._test_simple_base()
-
- @coroutine_test
- async def test_builtin_simple(self):
- await self._test_simple_base(self.MW_SIMPLE, start_index=1000)
+ await self._test_asyncgen_base()
@coroutine_test
async def test_builtin_async(self):
@@ -503,11 +402,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_builtin_universal(self):
- await self._test_simple_base(self.MW_UNIVERSAL, start_index=1000)
-
- @coroutine_test
- async def test_simple_builtin(self):
- await self._test_simple_base(self.MW_SIMPLE)
+ await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)
@coroutine_test
async def test_async_builtin(self):
@@ -516,7 +411,7 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_universal_builtin(self):
- await self._test_simple_base(self.MW_UNIVERSAL)
+ await self._test_asyncgen_base(self.MW_UNIVERSAL)
class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
@@ -524,40 +419,9 @@ class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
for item in super()._callback():
yield item
- @coroutine_test
- async def test_just_builtin(self):
- await self._test_asyncgen_base()
-
- @coroutine_test
- async def test_builtin_simple(self):
- """Downgrade"""
- await self._test_simple_base(self.MW_SIMPLE, downgrade=True, start_index=1000)
-
- @coroutine_test
- async def test_builtin_async(self):
- await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)
-
- @coroutine_test
- async def test_builtin_universal(self):
- await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)
-
- @coroutine_test
- async def test_simple_builtin(self):
- """Downgrade"""
- await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
-
- @coroutine_test
- async def test_async_builtin(self):
- await self._test_asyncgen_base(self.MW_ASYNCGEN)
-
- @coroutine_test
- async def test_universal_builtin(self):
- await self._test_asyncgen_base(self.MW_UNIVERSAL)
-
class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
ITEM_TYPE = dict
- MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
MW_EXC_SIMPLE = ProcessSpiderExceptionSimpleIterableMiddleware
@@ -576,18 +440,13 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_exc_simple(self):
"""Simple exc mw"""
- await self._test_simple_base(self.MW_EXC_SIMPLE)
+ await self._test_asyncgen_base(self.MW_EXC_SIMPLE)
@coroutine_test
async def test_exc_async(self):
"""Async exc mw"""
await self._test_asyncgen_base(self.MW_EXC_ASYNCGEN)
- @coroutine_test
- async def test_exc_simple_simple(self):
- """Simple exc mw -> simple output mw"""
- await self._test_simple_base(self.MW_SIMPLE, self.MW_EXC_SIMPLE)
-
@coroutine_test
async def test_exc_async_async(self):
"""Async exc mw -> async output mw"""
@@ -598,25 +457,27 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
"""Simple exc mw -> async output mw; upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_EXC_SIMPLE)
- @coroutine_test
- async def test_exc_async_simple(self):
- """Async exc mw -> simple output mw; cannot work as downgrading is not supported"""
- await self._test_asyncgen_nodowngrade(self.MW_SIMPLE, self.MW_EXC_ASYNCGEN)
-
class TestDeprecatedSpiderArg(TestSpiderMiddleware):
@coroutine_test
async def test_deprecated_mw_spider_arg(self):
- class DeprecatedSpiderArgMiddleware:
+ class DeprecatedSpiderArgMiddleware1:
def process_spider_input(self, response, spider):
return None
- def process_spider_output(self, response, result, spider):
+ async def process_spider_output(self, response, result, spider):
1 / 0
+ yield
+ class DeprecatedSpiderArgMiddleware2:
def process_spider_exception(self, response, exception, spider):
return []
+ with pytest.warns(
+ ScrapyDeprecationWarning,
+ match=r"process_spider_exception\(\) requires a spider argument",
+ ):
+ self.mwman._add_middleware(DeprecatedSpiderArgMiddleware2())
with (
pytest.warns(
ScrapyDeprecationWarning,
@@ -626,13 +487,10 @@ class TestDeprecatedSpiderArg(TestSpiderMiddleware):
ScrapyDeprecationWarning,
match=r"process_spider_output\(\) requires a spider argument",
),
- pytest.warns(
- ScrapyDeprecationWarning,
- match=r"process_spider_exception\(\) requires a spider argument",
- ),
):
- self.mwman._add_middleware(DeprecatedSpiderArgMiddleware())
- await self._scrape_response()
+ self.mwman._add_middleware(DeprecatedSpiderArgMiddleware1())
+ it = await self._scrape_response()
+ await collect_asyncgen(it)
@coroutine_test
async def test_deprecated_mwman_spider_arg(self):
diff --git a/tests/test_spidermiddleware_output_chain.py b/tests/test_spidermiddleware_output_chain.py
index 8cf08dd94..afdfaa322 100644
--- a/tests/test_spidermiddleware_output_chain.py
+++ b/tests/test_spidermiddleware_output_chain.py
@@ -169,8 +169,8 @@ class NotGeneratorCallbackSpiderMiddlewareRightAfterSpider(NotGeneratorCallbackS
# ================================================================================
# (4) exceptions from a middleware process_spider_output method (generator)
class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- for r in result:
+ async def process_spider_output(self, response, result):
+ async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
@@ -182,8 +182,8 @@ class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
class GeneratorFailMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- for r in result:
+ async def process_spider_output(self, response, result):
+ async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
raise LookupError
@@ -201,8 +201,8 @@ class GeneratorDoNothingAfterFailureMiddleware(_GeneratorDoNothingMiddleware):
class GeneratorRecoverMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- for r in result:
+ async def process_spider_output(self, response, result):
+ async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
@@ -237,86 +237,6 @@ class GeneratorOutputChainSpider(Spider):
yield {"processed": ["parse-second-item"]}
-# ================================================================================
-# (5) exceptions from a middleware process_spider_output method (not generator)
-
-
-class _NotGeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- out = []
- for r in result:
- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
- out.append(r)
- return out
-
- def process_spider_exception(self, response, exception):
- method = f"{self.__class__.__name__}.process_spider_exception"
- self.crawler.spider.logger.info(
- "%s: %s caught", method, exception.__class__.__name__
- )
-
-
-class NotGeneratorFailMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- out = []
- for r in result:
- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
- out.append(r)
- raise ReferenceError
-
- def process_spider_exception(self, response, exception):
- method = f"{self.__class__.__name__}.process_spider_exception"
- self.crawler.spider.logger.info(
- "%s: %s caught", method, exception.__class__.__name__
- )
- return [{"processed": [method]}]
-
-
-class NotGeneratorDoNothingAfterFailureMiddleware(_NotGeneratorDoNothingMiddleware):
- pass
-
-
-class NotGeneratorRecoverMiddleware(_BaseSpiderMiddleware):
- def process_spider_output(self, response, result):
- out = []
- for r in result:
- r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
- out.append(r)
- return out
-
- def process_spider_exception(self, response, exception):
- method = f"{self.__class__.__name__}.process_spider_exception"
- self.crawler.spider.logger.info(
- "%s: %s caught", method, exception.__class__.__name__
- )
- return [{"processed": [method]}]
-
-
-class NotGeneratorDoNothingAfterRecoveryMiddleware(_NotGeneratorDoNothingMiddleware):
- pass
-
-
-class NotGeneratorOutputChainSpider(Spider):
- name = "NotGeneratorOutputChainSpider"
- custom_settings = {
- "SPIDER_MIDDLEWARES": {
- NotGeneratorFailMiddleware: 10,
- NotGeneratorDoNothingAfterFailureMiddleware: 8,
- NotGeneratorRecoverMiddleware: 5,
- NotGeneratorDoNothingAfterRecoveryMiddleware: 3,
- },
- }
-
- async def start(self):
- yield Request(self.mockserver.url("/status?n=200"))
-
- def parse(self, response):
- return [
- {"processed": ["parse-first-item"]},
- {"processed": ["parse-second-item"]},
- ]
-
-
# ================================================================================
class TestSpiderMiddleware:
mockserver: MockServer
@@ -481,41 +401,3 @@ class TestSpiderMiddleware:
assert str(item_from_callback) in str(log4)
assert str(item_recovered) in str(log4)
assert "parse-second-item" not in str(log4)
-
- @coroutine_test
- async def test_not_a_generator_output_chain(self):
- """
- (5) An exception from a middleware's process_spider_output method should be sent
- to the process_spider_exception method from the next middleware in the chain.
- The result of the recovery by the process_spider_exception method should be handled
- by the process_spider_output method from the next middleware.
- The final item count should be 1 (from the process_spider_exception chain, the items
- from the spider callback are lost)
- """
- log5 = await self.crawl_log(NotGeneratorOutputChainSpider)
- assert "'item_scraped_count': 1" in str(log5)
- assert (
- "GeneratorRecoverMiddleware.process_spider_exception: ReferenceError caught"
- in str(log5)
- )
- assert (
- "GeneratorDoNothingAfterFailureMiddleware.process_spider_exception: ReferenceError caught"
- in str(log5)
- )
- assert (
- "GeneratorFailMiddleware.process_spider_exception: ReferenceError caught"
- not in str(log5)
- )
- assert (
- "GeneratorDoNothingAfterRecoveryMiddleware.process_spider_exception: ReferenceError caught"
- not in str(log5)
- )
- item_recovered = {
- "processed": [
- "NotGeneratorRecoverMiddleware.process_spider_exception",
- "NotGeneratorDoNothingAfterRecoveryMiddleware.process_spider_output",
- ]
- }
- assert str(item_recovered) in str(log5)
- assert "parse-first-item" not in str(log5)
- assert "parse-second-item" not in str(log5)
diff --git a/tests/test_utils_python.py b/tests/test_utils_python.py
index 8dfac5f02..e8fe45749 100644
--- a/tests/test_utils_python.py
+++ b/tests/test_utils_python.py
@@ -12,7 +12,6 @@ from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
from scrapy.utils.defer import aiter_errback
from scrapy.utils.python import (
MutableAsyncChain,
- MutableChain,
binary_is_text,
get_func_args,
memoizemethod_noargs,
@@ -30,16 +29,6 @@ _KT = TypeVar("_KT")
_VT = TypeVar("_VT")
-def test_mutablechain():
- m = MutableChain(range(2), [2, 3], (4, 5))
- m.extend(range(6, 7))
- m.extend([7, 8])
- m.extend([9, 10], (11, 12))
- assert next(m) == 0
- assert m.__next__() == 1
- assert list(m) == list(range(2, 13))
-
-
class TestMutableAsyncChain:
@staticmethod
async def g1():