Scrapy, a fast high-level web crawling & scraping framework for Python.
Implement spider middleware iterable upgrade/downgrade.
Andrey Rakhmatullinf789547551ae
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex c514fc00a..073b6bd9a 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -35,11 +35,10 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- :ref:`Signal handlers that support deferreds <signal-deferred>`.- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`- method of :ref:`spider middlewares <custom-spider-middleware>`.+ method of :ref:`spider middlewares <custom-spider-middleware>`. See+ :ref:`async-spider-middlewares`... versionadded:: VERSION- .. note:: This method needs to be an async generator, not just a coroutine that- returns an iterable.Usage=====@@ -106,8 +105,8 @@ Common use cases for asynchronous code include:* storing data in databases (in pipelines and middlewares);* delaying the spider initialization until some external event (in the:signal:`spider_opened` handler);-* calling asynchronous Scrapy methods like ``ExecutionEngine.download`` (see- :ref:`the screenshot pipeline example<ScreenshotPipeline>`).+* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download`+ (see :ref:`the screenshot pipeline example<ScreenshotPipeline>`)... _aio-libs: https://github.com/aio-libs@@ -119,59 +118,72 @@ Asynchronous spider middlewares.. versionadded:: VERSION.. note:: This currently applies to:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`.+ In the future it will also apply to+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`.Middleware methods discussed here can take and return async iterables. They canreturn the same type of iterable or they can take a normal one and return anasync one. If such method needs to return an async iterable it must be an asyncgenerator, not just a coroutine that returns an iterable.-.. autofunction:: scrapy.utils.asyncgen.as_async_generator--In the simplest form that supports both sync and async input it can be written-like this::-- from scrapy.utils.asyncgen import as_async_generator-- class ProcessSpiderOutputAsyncGenMiddleware:- async def process_spider_output(self, response, result, spider):- async for r in as_async_generator(result):+As the result of a middleware method is passed to the same method of the next+middleware, it needs to be adapted if the second method expects a different+type. Scrapy will do this transparently:++* A normal iterable is wrapped into an async one which shouldn't cause any side+ effects.+* An async iterable is downgraded to a normal one by waiting until all results+ are available and wrapping them in a normal iterable. This is problematic+ because it pauses the normal middleware processing for this iterable and+ because all results can be skipped if exceptions are raised during+ processing. This case emits a warning and will be deprecated and then removed+ in a later Scrapy version.+* Async iterables returned from+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`+ won't be downgraded, an exception will be raised if that is needed.++As downgrading is undesirable, here is the proposed way to avoid it. If all+middlewares, including 3rd-party ones, support async iterables as input, no+downgrading will happen. But removing normal iterable support (making the+method a coroutine) from a middleware published as a separate project or used+internally in projects for older Scrapy versions breaks backwards+compatibility. So, as an interim measure (it will be deprecated and then+removed in a later Scrapy version), a middleware can provide both sync and+async methods in the following form::++ class UniversalSpiderMiddleware:+ def process_spider_output(self, response, result, spider):+ for r in result:# ... do something with ryield r-If the middleware input (the callback result for ``process_spider_output``) is-an async iterable, all middlewares that process it must support it. The-built-in ones do, but the ones in your project and 3rd-party ones will need to-be updated to support it, as the code that expects a normal iterable will break-on an async one. If these middlewares receive an async iterable, they must-return one as well. On the other hand, if they receive a normal iterable, they-shouldn't break and ideally should return a normal iterable too. There can be-several possible implementations of this.--The simplest one, always converting normal iterables to async ones, is provided-above. Because a result of a middleware method is passed to the same method of-the next middleware, it's only possible to mix middlewares with synchronous and-asynchronous implementations of the same method if all synchronous ones are-called first (which isn't always possible).--Another option is to make separate methods for normal and async iterables and-choose one at run time::-- from inspect import isasyncgen-- class ProcessSpiderOutputAsyncGenMiddleware:- def _normal_process_spider_output(self, response, result, spider):- # ... do something with normal result-- async def _async_process_spider_output(self, response, result, spider):- # ... do the same with async result+ async def process_spider_output_async(self, response, result, spider):+ async for r in result:+ # ... do something with r+ yield r- def process_spider_output(self, response, result, spider):- if isasyncgen(result):- return self._async_process_spider_output(self, response, result, spider)- else:- return self._normal_process_spider_output(self, response, result, spider)--If you are writing a middleware that you intend to publish or to use in many-projects, this is likely the best way to implement it. It may be possible to-extract common code from both methods to reduce code duplication, as in the-simplest case the only difference between them will be ``for`` vs ``async for``.+In this case normal and async iterables will be passed to the respective+methods without any wrapping or downgrading, and in older versions of Scrapy+the coroutine method will just be ignored. When the backwards compatibility is+no longer needed the non-coroutine method can be dropped and the coroutine one+renamed to the normal name. It may be possible to extract common code from both+methods to reduce code duplication, as in the simplest case the only difference+between them will be ``for`` vs ``async for``.++So, to recap:++* If you don't intend to use async callbacks or middlewares containing async+ code in your project, nothing should change for you yet. At some point in the+ future some of the 3rd-party middlewares you use may drop backwards+ compatibility, which shouldn't lead to immediate problems but may be a sign+ to start converting your code to ``async def`` too.+* If you maintain a middleware that can be used with projects you can't control+ (e.g. one you published for other people to use, or one that needs to support+ some old project that can't be modernized), we recommend adding a+ ``process_spider_output_async`` method so that the amount of unnecessary+ iterable conversions is reduced but no compatibility is broken.+* If you use async callbacks, try to make sure all middlewares support them.+ Note that you can modernize 3rd-party middlewares by subclassing them.+* If you want to write and publish a middleware that requires async code, you+ should write in the docs that the minimum support Scrapy version is VERSION+ (maybe even check this at the run time, using :attr:`scrapy.__version__`).diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex f8d4a356f..edfc2e4bb 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -123,6 +123,15 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.:param spider: the spider whose result is being processed:type spider: :class:`~scrapy.Spider` object+ .. method:: process_spider_output_async(response, result, spider)++ .. versionadded:: VERSION++ If exists, this methid will be called instead of+ :meth:`process_spider_output` when ``result`` is an async iterable.+ If this method exists, it must be a coroutine while+ :meth:`process_spider_output` must not be a coroutine.+.. method:: process_spider_exception(response, exception, spider)This method is called when a spider or :meth:`process_spider_output`diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.pyindex e64cfae0a..e1fdd8d13 100644--- a/scrapy/core/scraper.py+++ b/scrapy/core/scraper.py@@ -192,7 +192,8 @@ class Scraper:spider=spider)- def handle_spider_output(self, result: Iterable, request: Request, response: Response, spider: Spider) -> Deferred:+ def handle_spider_output(self, result: Union[Iterable, AsyncIterable], request: Request,+ response: Response, spider: Spider) -> Deferred:if not result:return defer_succeed(None)it: Union[Generator, AsyncGenerator]diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.pyindex 009ece06d..6075670b0 100644--- a/scrapy/core/spidermw.py+++ b/scrapy/core/spidermw.py@@ -3,22 +3,27 @@ Spider Middleware managerSee documentation in docs/topics/spider-middleware.rst"""+import logging+from inspect import isasyncgenfunctionfrom itertools import islice-from typing import Any, AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Union, cast+from typing import Any, AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Tuple, Union, cast-from twisted.internet.defer import Deferred+from twisted.internet.defer import Deferred, inlineCallbacksfrom twisted.python.failure import Failurefrom scrapy import Request, Spiderfrom scrapy.exceptions import _InvalidOutputfrom scrapy.http import Responsefrom scrapy.middleware import MiddlewareManager-from scrapy.utils.asyncgen import _process_iterable_universal+from scrapy.utils.asyncgen import as_async_generator, collect_asyncgenfrom scrapy.utils.conf import build_component_list-from scrapy.utils.defer import mustbe_deferred+from scrapy.utils.defer import mustbe_deferred, deferred_from_coro, deferred_f_from_coro_f, maybe_deferred_to_futurefrom scrapy.utils.python import MutableAsyncChain, MutableChain+logger = logging.getLogger(__name__)++ScrapeFunc = Callable[[Union[Response, Failure], Request, Spider], Any]@@ -30,6 +35,10 @@ class SpiderMiddlewareManager(MiddlewareManager):component_name = 'spider middleware'+ def __init__(self, *middlewares):+ super().__init__(*middlewares)+ self.downgrade_warning_done = False+@classmethoddef _get_mwlist_from_settings(cls, settings):return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES'))@@ -40,7 +49,7 @@ class SpiderMiddlewareManager(MiddlewareManager):self.methods['process_spider_input'].append(mw.process_spider_input)if hasattr(mw, 'process_start_requests'):self.methods['process_start_requests'].appendleft(mw.process_start_requests)- process_spider_output = getattr(mw, 'process_spider_output', None)+ process_spider_output = self._get_async_method_pair(mw, 'process_spider_output')self.methods['process_spider_output'].appendleft(process_spider_output)process_spider_exception = getattr(mw, 'process_spider_exception', None)self.methods['process_spider_exception'].appendleft(process_spider_exception)@@ -64,8 +73,19 @@ class SpiderMiddlewareManager(MiddlewareManager):def _evaluate_iterable(self, response: Response, spider: Spider, iterable: Union[Iterable, AsyncIterable],exception_processor_index: int, recover_to: Union[MutableChain, MutableAsyncChain]) -> Union[Generator, AsyncGenerator]:- @_process_iterable_universal- async def _evaluate_async_iterable(iterable):++ def process_sync(iterable: Iterable):+ try:+ for r in iterable:+ yield r+ except Exception as ex:+ exception_result = self._process_spider_exception(response, spider, Failure(ex),+ exception_processor_index)+ if isinstance(exception_result, Failure):+ raise+ recover_to.extend(exception_result)++ async def process_async(iterable: AsyncIterable):try:async for r in iterable:yield r@@ -75,7 +95,10 @@ class SpiderMiddlewareManager(MiddlewareManager):if isinstance(exception_result, Failure):raiserecover_to.extend(exception_result)- return _evaluate_async_iterable(iterable)++ if isinstance(iterable, AsyncIterable):+ return process_async(iterable)+ return process_sync(iterable)def _process_spider_exception(self, response: Response, spider: Spider, _failure: Failure,start_index: int = 0) -> Union[Failure, MutableChain]:@@ -87,11 +110,22 @@ class SpiderMiddlewareManager(MiddlewareManager):for method_index, method in enumerate(method_list, start=start_index):if method is None:continue+ method = cast(Callable, method)result = method(response=response, exception=exception, spider=spider)if _isiterable(result):# stop exception handling by handing control over to the# process_spider_output chain if an iterable has been returned- return self._process_spider_output(response, spider, result, method_index + 1)+ dfd: Deferred = self._process_spider_output(response, spider, result, method_index + 1)+ # _process_spider_output() returns a Deferred only because of downgrading so this can be+ # simplified when downgrading is removed.+ if dfd.called:+ # the result is available immediately if _process_spider_output didn't do downgrading+ return dfd.result+ else:+ # we forbid waiting here because otherwise we would need to return a deferred from+ # _process_spider_exception too, which complicates the architecture+ msg = f"Async iterable returned from {method.__qualname__} cannot be downgraded"+ raise _InvalidOutput(msg)elif result is None:continueelse:@@ -100,9 +134,13 @@ class SpiderMiddlewareManager(MiddlewareManager):raise _InvalidOutput(msg)return _failure+ # This method cannot be made async def, as _process_spider_exception relies on the Deferred result+ # being available immediately which doesn't work when it's a wrapped coroutine.+ # It also needs @inlineCallbacks only because of downgrading so it can be removed when downgrading is removed.+ @inlineCallbacksdef _process_spider_output(self, response: Response, spider: Spider,result: Union[Iterable, AsyncIterable], start_index: int = 0- ) -> Union[MutableChain, MutableAsyncChain]:+ ) -> Deferred:# items in this iterable do not need to go through the process_spider_output# chain, they went through it already from the process_spider_exception methodrecovered: Union[MutableChain, MutableAsyncChain]@@ -112,11 +150,43 @@ class SpiderMiddlewareManager(MiddlewareManager):else:recovered = MutableChain()+ # There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.+ # 1. def foo. Sync iterables are passed as is, async ones are downgraded.+ # 2. async def foo. Sync iterables are upgraded, async ones are passed as is.+ # 3. def foo + async def foo_async. Iterables are passed to the respective method.+ # Storing methods and method tuples in the same list is weird but we should be able to roll this back+ # when we drop this compatibility feature.+method_list = islice(self.methods['process_spider_output'], start_index, None)- for method_index, method in enumerate(method_list, start=start_index):- if method is None:+ for method_index, method_pair in enumerate(method_list, start=start_index):+ if method_pair is None:continue+ need_upgrade = need_downgrade = False+ if isinstance(method_pair, tuple):+ # This tuple handling is only needed until _async compatibility methods are removed.+ method_sync, method_async = method_pair+ method = method_async if last_result_is_async else method_sync+ else:+ method = method_pair+ if not last_result_is_async and isasyncgenfunction(method):+ need_upgrade = True+ elif last_result_is_async and not isasyncgenfunction(method):+ need_downgrade = Truetry:+ if need_upgrade:+ # Iterable -> AsyncIterable+ result = as_async_generator(result)+ elif need_downgrade:+ if not self.downgrade_warning_done:+ logger.warning(f"Async iterable passed to {method.__qualname__} "+ f"was downgraded to a non-async one")+ self.downgrade_warning_done = True+ assert isinstance(result, AsyncIterable)+ # AsyncIterable -> Iterable+ result = yield deferred_from_coro(collect_asyncgen(result))+ if isinstance(recovered, AsyncIterable):+ recovered_collected = yield deferred_from_coro(collect_asyncgen(recovered))+ recovered = MutableChain(recovered_collected)# might fail directly if the output value is not a generatorresult = method(response=response, result=result, spider=spider)except Exception as ex:@@ -130,8 +200,6 @@ class SpiderMiddlewareManager(MiddlewareManager):msg = (f"Middleware {method.__qualname__} must return an "f"iterable, got {type(result)}")raise _InvalidOutput(msg)- if last_result_is_async and isinstance(result, Iterable):- raise TypeError(f"Synchronous {method.__qualname__} called with an async iterable")last_result_is_async = isinstance(result, AsyncIterable)if last_result_is_async:@@ -139,31 +207,58 @@ class SpiderMiddlewareManager(MiddlewareManager):else:return MutableChain(result, recovered) # type: ignore[arg-type]- def _process_callback_output(self, response: Response, spider: Spider, result: Union[Iterable, AsyncIterable]- ) -> Union[MutableChain, MutableAsyncChain]:+ async def _process_callback_output(self, response: Response, spider: Spider, result: Union[Iterable, AsyncIterable]+ ) -> Union[MutableChain, MutableAsyncChain]:recovered: Union[MutableChain, MutableAsyncChain]if isinstance(result, AsyncIterable):recovered = MutableAsyncChain()else:recovered = MutableChain()result = self._evaluate_iterable(response, spider, result, 0, recovered)- result = self._process_spider_output(response, spider, result)+ result = await maybe_deferred_to_future(self._process_spider_output(response, spider, result))if isinstance(result, AsyncIterable):return MutableAsyncChain(result, recovered)else:+ if isinstance(recovered, AsyncIterable):+ recovered_collected = await collect_asyncgen(recovered)+ recovered = MutableChain(recovered_collected)return MutableChain(result, recovered) # type: ignore[arg-type]def scrape_response(self, scrape_func: ScrapeFunc, response: Response, request: Request,spider: Spider) -> Deferred:- def process_callback_output(result: Union[Iterable, AsyncIterable]) -> Union[MutableChain, MutableAsyncChain]:- return self._process_callback_output(response, spider, result)+ async def process_callback_output(result: Union[Iterable, AsyncIterable]+ ) -> Union[MutableChain, MutableAsyncChain]:+ return await self._process_callback_output(response, spider, result)def process_spider_exception(_failure: Failure) -> Union[Failure, MutableChain]:return self._process_spider_exception(response, spider, _failure)dfd = mustbe_deferred(self._process_spider_input, scrape_func, response, request, spider)- dfd.addCallbacks(callback=process_callback_output, errback=process_spider_exception)+ dfd.addCallbacks(callback=deferred_f_from_coro_f(process_callback_output), errback=process_spider_exception)return dfddef process_start_requests(self, start_requests, spider: Spider) -> Deferred:return self._process_chain('process_start_requests', start_requests, spider)++ # This method is only needed until _async compatibility methods are removed.+ @staticmethod+ def _get_async_method_pair(mw: Any, methodname: str) -> Union[None, Callable, Tuple[Callable, Callable]]:+ normal_method = getattr(mw, methodname, None)+ methodname_async = methodname + "_async"+ async_method = getattr(mw, methodname_async, None)+ if not async_method:+ return normal_method+ if not normal_method:+ logger.error(f"Middleware {mw.__qualname__} has {methodname_async} "+ f"without {methodname}, skipping this method.")+ return None+ if not isasyncgenfunction(async_method):+ logger.error(f"{async_method.__qualname__} is not "+ f"an async generator function, skipping this method.")+ return normal_method+ if isasyncgenfunction(normal_method):+ logger.error(f"{normal_method.__qualname__} is an async "+ f"generator function while {methodname_async} exists, "+ f"skipping both methods.")+ return None+ return normal_method, async_methoddiff --git a/scrapy/middleware.py b/scrapy/middleware.pyindex 2eb1d8609..8d7e5a602 100644--- a/scrapy/middleware.py+++ b/scrapy/middleware.py@@ -1,7 +1,7 @@import loggingimport pprintfrom collections import defaultdict, deque-from typing import Callable, Deque, Dict, Optional, cast, Iterable+from typing import Callable, Deque, Dict, Iterable, Tuple, Union, castfrom twisted.internet.defer import Deferred@@ -21,8 +21,9 @@ class MiddlewareManager:def __init__(self, *middlewares):self.middlewares = middlewares- # Optional because process_spider_output and process_spider_exception can be None- self.methods: Dict[str, Deque[Optional[Callable]]] = defaultdict(deque)+ # Only process_spider_output and process_spider_exception can be None.+ # Only process_spider_output can be a tuple, and only until _async compatibility methods are removed.+ self.methods: Dict[str, Deque[Union[None, Callable, Tuple[Callable, Callable]]]] = defaultdict(deque)for mw in middlewares:self._add_middleware(mw)diff --git a/scrapy/spidermiddlewares/depth.py b/scrapy/spidermiddlewares/depth.pyindex 973404b2b..29634c3ad 100644--- a/scrapy/spidermiddlewares/depth.py+++ b/scrapy/spidermiddlewares/depth.py@@ -7,7 +7,6 @@ See documentation in docs/topics/spider-middleware.rstimport loggingfrom scrapy.http import Request-from scrapy.utils.asyncgen import _process_iterable_universallogger = logging.getLogger(__name__)@@ -29,36 +28,42 @@ class DepthMiddleware:return cls(maxdepth, crawler.stats, verbose, prio)def process_spider_output(self, response, result, spider):- def _filter(request):- if isinstance(request, Request):- depth = response.meta['depth'] + 1- request.meta['depth'] = depth- if self.prio:- request.priority -= depth * self.prio- if self.maxdepth and depth > self.maxdepth:- logger.debug(- "Ignoring link (depth > %(maxdepth)d): %(requrl)s ",- {'maxdepth': self.maxdepth, 'requrl': request.url},- extra={'spider': spider}- )- return False- else:- if self.verbose_stats:- self.stats.inc_value(f'request_depth_count/{depth}',- spider=spider)- self.stats.max_value('request_depth_max', depth,- spider=spider)- return True+ # base case (depth=0)+ if 'depth' not in response.meta:+ response.meta['depth'] = 0+ if self.verbose_stats:+ self.stats.inc_value('request_depth_count/0', spider=spider)++ return (r for r in result or () if self._filter(r, response, spider))- @_process_iterable_universal- async def process(result):- # base case (depth=0)- if 'depth' not in response.meta:- response.meta['depth'] = 0- if self.verbose_stats:- self.stats.inc_value('request_depth_count/0', spider=spider)+ async def process_spider_output_async(self, response, result, spider):+ # base case (depth=0)+ if 'depth' not in response.meta:+ response.meta['depth'] = 0+ if self.verbose_stats:+ self.stats.inc_value('request_depth_count/0', spider=spider)- async for r in result or ():- if _filter(r):- yield r- return process(result)+ async for r in result or ():+ if self._filter(r, response, spider):+ yield r++ def _filter(self, request, response, spider):+ if not isinstance(request, Request):+ return True+ depth = response.meta['depth'] + 1+ request.meta['depth'] = depth+ if self.prio:+ request.priority -= depth * self.prio+ if self.maxdepth and depth > self.maxdepth:+ logger.debug(+ "Ignoring link (depth > %(maxdepth)d): %(requrl)s ",+ {'maxdepth': self.maxdepth, 'requrl': request.url},+ extra={'spider': spider}+ )+ return False+ if self.verbose_stats:+ self.stats.inc_value(f'request_depth_count/{depth}',+ spider=spider)+ self.stats.max_value('request_depth_max', depth,+ spider=spider)+ return Truediff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.pyindex 074ec7a4e..448bc1367 100644--- a/scrapy/spidermiddlewares/offsite.py+++ b/scrapy/spidermiddlewares/offsite.py@@ -9,7 +9,6 @@ import warningsfrom scrapy import signalsfrom scrapy.http import Request-from scrapy.utils.asyncgen import _process_iterable_universalfrom scrapy.utils.httpobj import urlparse_cachedlogger = logging.getLogger(__name__)@@ -27,24 +26,27 @@ class OffsiteMiddleware:return odef process_spider_output(self, response, result, spider):- @_process_iterable_universal- async def process(result):- async for x in result:- if isinstance(x, Request):- if x.dont_filter or self.should_follow(x, spider):- yield x- else:- domain = urlparse_cached(x).hostname- if domain and domain not in self.domains_seen:- self.domains_seen.add(domain)- logger.debug(- "Filtered offsite request to %(domain)r: %(request)s",- {'domain': domain, 'request': x}, extra={'spider': spider})- self.stats.inc_value('offsite/domains', spider=spider)- self.stats.inc_value('offsite/filtered', spider=spider)- else:- yield x- return process(result)+ return (r for r in result or () if self._filter(r, spider))++ async def process_spider_output_async(self, response, result, spider):+ async for r in result or ():+ if self._filter(r, spider):+ yield r++ def _filter(self, request, spider) -> bool:+ if not isinstance(request, Request):+ return True+ if request.dont_filter or self.should_follow(request, spider):+ return True+ domain = urlparse_cached(request).hostname+ if domain and domain not in self.domains_seen:+ self.domains_seen.add(domain)+ logger.debug(+ "Filtered offsite request to %(domain)r: %(request)s",+ {'domain': domain, 'request': request}, extra={'spider': spider})+ self.stats.inc_value('offsite/domains', spider=spider)+ self.stats.inc_value('offsite/filtered', spider=spider)+ return Falsedef should_follow(self, request, spider):regex = self.host_regexdiff --git a/scrapy/spidermiddlewares/referer.py b/scrapy/spidermiddlewares/referer.pyindex e0a22592f..8027beb92 100644--- a/scrapy/spidermiddlewares/referer.py+++ b/scrapy/spidermiddlewares/referer.py@@ -11,7 +11,6 @@ from w3lib.url import safe_url_stringfrom scrapy import signalsfrom scrapy.exceptions import NotConfiguredfrom scrapy.http import Request, Response-from scrapy.utils.asyncgen import _process_iterable_universalfrom scrapy.utils.misc import load_objectfrom scrapy.utils.python import to_unicodefrom scrapy.utils.url import strip_url@@ -334,19 +333,18 @@ class RefererMiddleware:return cls() if cls else self.default_policy()def process_spider_output(self, response, result, spider):- def _set_referer(r):- if isinstance(r, Request):- referrer = self.policy(response, r).referrer(response.url, r.url)- if referrer is not None:- r.headers.setdefault('Referer', referrer)- return r-- @_process_iterable_universal- async def process(result):- async for r in result or ():- yield _set_referer(r)-- return process(result)+ return (self._set_referer(r, response) for r in result or ())++ async def process_spider_output_async(self, response, result, spider):+ async for r in result or ():+ yield self._set_referer(r, response)++ def _set_referer(self, r, response):+ if isinstance(r, Request):+ referrer = self.policy(response, r).referrer(response.url, r.url)+ if referrer is not None:+ r.headers.setdefault('Referer', referrer)+ return rdef request_scheduled(self, request, spider):# check redirected request to patch "Referer" header if necessarydiff --git a/scrapy/spidermiddlewares/urllength.py b/scrapy/spidermiddlewares/urllength.pyindex 63b1d36bd..7ad64d2af 100644--- a/scrapy/spidermiddlewares/urllength.py+++ b/scrapy/spidermiddlewares/urllength.py@@ -8,7 +8,6 @@ import loggingfrom scrapy.http import Requestfrom scrapy.exceptions import NotConfigured-from scrapy.utils.asyncgen import _process_iterable_universallogger = logging.getLogger(__name__)@@ -26,22 +25,20 @@ class UrlLengthMiddleware:return cls(maxlength)def process_spider_output(self, response, result, spider):- def _filter(request):- if isinstance(request, Request) and len(request.url) > self.maxlength:- logger.info(- "Ignoring link (url length > %(maxlength)d): %(url)s ",- {'maxlength': self.maxlength, 'url': request.url},- extra={'spider': spider}- )- spider.crawler.stats.inc_value('urllength/request_ignored_count', spider=spider)- return False- else:- return True-- @_process_iterable_universal- async def process(result):- async for r in result or ():- if _filter(r):- yield r-- return process(result)+ return (r for r in result or () if self._filter(r, spider))++ async def process_spider_output_async(self, response, result, spider):+ async for r in result or ():+ if self._filter(r, spider):+ yield r++ def _filter(self, request, spider):+ if isinstance(request, Request) and len(request.url) > self.maxlength:+ logger.info(+ "Ignoring link (url length > %(maxlength)d): %(url)s ",+ {'maxlength': self.maxlength, 'url': request.url},+ extra={'spider': spider}+ )+ spider.crawler.stats.inc_value('urllength/request_ignored_count', spider=spider)+ return False+ return Truediff --git a/scrapy/utils/asyncgen.py b/scrapy/utils/asyncgen.pyindex ae9a79989..9f794de92 100644--- a/scrapy/utils/asyncgen.py+++ b/scrapy/utils/asyncgen.py@@ -1,6 +1,4 @@-import functools-import inspect-from typing import AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Union+from typing import AsyncGenerator, AsyncIterable, Iterable, Unionasync def collect_asyncgen(result: AsyncIterable):@@ -18,46 +16,3 @@ async def as_async_generator(it: Union[Iterable, AsyncIterable]) -> AsyncGeneratelse:for r in it:yield r---# https://stackoverflow.com/a/66170760/113586-def _process_iterable_universal(process_async: Callable):- """ Takes a function that takes an async iterable, args and kwargs. Returns- a function that takes any iterable, args and kwargs.-- Requires that process_async only awaits on the iterable and synchronous functions,- so it's better to use this only in the Scrapy code itself.- """-- # If this stops working, all internal uses can be just replaced with manually-written- # process_sync functions.-- def process_sync(iterable: Iterable, *args, **kwargs) -> Generator:- agen = process_async(as_async_generator(iterable), *args, **kwargs)- if not inspect.isasyncgen(agen):- raise ValueError(f"process_async returned wrong type {type(agen)}")- sent = None- while True:- try:- gen = agen.asend(sent)- gen.send(None)- except StopIteration as e:- sent = yield e.value- except StopAsyncIteration:- return- else:- gen.throw(RuntimeError,- f"Synchronously-called function '{process_async.__name__}' has blocked, "- f"you can't use {_process_iterable_universal.__name__} with it.")-- @functools.wraps(process_async)- def process(iterable: Union[Iterable, AsyncIterable], *args, **kwargs) -> Union[Generator, AsyncGenerator]:- if isinstance(iterable, AsyncIterable):- # call process_async directly- return process_async(iterable, *args, **kwargs)- if isinstance(iterable, Iterable):- # convert process_async to process_sync- return process_sync(iterable, *args, **kwargs)- raise TypeError(f"Wrong iterable type {type(iterable)}")-- return processdiff --git a/scrapy/utils/python.py b/scrapy/utils/python.pyindex d086347bc..11c089ac2 100644--- a/scrapy/utils/python.py+++ b/scrapy/utils/python.py@@ -10,7 +10,7 @@ import warningsimport weakreffrom functools import partial, wrapsfrom itertools import chain-from typing import AsyncIterable, Iterable, Union+from typing import AsyncGenerator, AsyncIterable, Iterable, Unionfrom scrapy.exceptions import ScrapyDeprecationWarningfrom scrapy.utils.asyncgen import as_async_generator@@ -359,7 +359,7 @@ class MutableChain(Iterable):return self.__next__()-async def _async_chain(*iterables: Union[Iterable, AsyncIterable]):+async def _async_chain(*iterables: Union[Iterable, AsyncIterable]) -> AsyncGenerator:for it in iterables:async for o in as_async_generator(it):yield odiff --git a/tests/test_spidermiddleware.py b/tests/test_spidermiddleware.pyindex b0ca2f62e..f9f2b6642 100644--- a/tests/test_spidermiddleware.py+++ b/tests/test_spidermiddleware.py@@ -1,6 +1,8 @@import collections.abc+from typing import Optionalfrom unittest import mock+from testfixtures import LogCapturefrom twisted.internet import deferfrom twisted.trial.unittest import TestCasefrom twisted.python.failure import Failure@@ -8,8 +10,8 @@ from twisted.python.failure import Failurefrom scrapy.spiders import Spiderfrom scrapy.http import Request, Responsefrom scrapy.exceptions import _InvalidOutput-from scrapy.utils.asyncgen import _process_iterable_universal, as_async_generator, collect_asyncgen-from scrapy.utils.defer import deferred_from_coro+from scrapy.utils.asyncgen import collect_asyncgen+from scrapy.utils.defer import deferred_from_coro, maybe_deferred_to_futurefrom scrapy.utils.test import get_crawlerfrom scrapy.core.spidermw import SpiderMiddlewareManager@@ -115,33 +117,40 @@ class BaseAsyncSpiderMiddlewareTestCase(SpiderMiddlewareTestCase):RESULT_COUNT = 3 # to simplify checks, let everything return 3 objects+ @staticmethod+ def _construct_mw_setting(*mw_classes, start_index: Optional[int] = None):+ if start_index is None:+ start_index = 10+ return {i: c for c, i in enumerate(mw_classes, start=start_index)}+@defer.inlineCallbacks- def _get_middleware_result(self, *mw_classes):- for mw_cls in mw_classes:- self.mwman._add_middleware(mw_cls())+ def _get_middleware_result(self, *mw_classes, start_index: Optional[int] = None):+ setting = self._construct_mw_setting(*mw_classes, start_index=start_index)+ self.crawler = get_crawler(Spider, {'SPIDER_MIDDLEWARES_BASE': {}, 'SPIDER_MIDDLEWARES': setting})+ self.spider = self.crawler._create_spider('foo')+ self.mwman = SpiderMiddlewareManager.from_crawler(self.crawler)result = yield self.mwman.scrape_response(self._scrape_func, self.response, self.request, self.spider)return result@defer.inlineCallbacks- def _test_simple_base(self, *mw_classes):- result = yield self._get_middleware_result(*mw_classes)+ def _test_simple_base(self, *mw_classes, downgrade: bool = False, start_index: Optional[int] = None):+ with LogCapture() as log:+ result = yield self._get_middleware_result(*mw_classes, start_index=start_index)self.assertIsInstance(result, collections.abc.Iterable)result_list = list(result)self.assertEqual(len(result_list), self.RESULT_COUNT)self.assertIsInstance(result_list[0], self.ITEM_TYPE)+ self.assertEqual("downgraded to a non-async" in str(log), downgrade)@defer.inlineCallbacks- def _test_asyncgen_base(self, *mw_classes):- result = yield self._get_middleware_result(*mw_classes)+ def _test_asyncgen_base(self, *mw_classes, downgrade: bool = False, start_index: Optional[int] = None):+ with LogCapture() as log:+ result = yield self._get_middleware_result(*mw_classes, start_index=start_index)self.assertIsInstance(result, collections.abc.AsyncIterator)result_list = yield deferred_from_coro(collect_asyncgen(result))self.assertEqual(len(result_list), self.RESULT_COUNT)self.assertIsInstance(result_list[0], self.ITEM_TYPE)-- @defer.inlineCallbacks- def _test_asyncgen_fail(self, *mw_classes):- with self.assertRaisesRegex(TypeError, "Synchronous .+ called with an async iterable"):- yield self._get_middleware_result(*mw_classes)+ self.assertEqual("downgraded to a non-async" in str(log), downgrade)class ProcessSpiderOutputSimpleMiddleware:@@ -152,17 +161,36 @@ class ProcessSpiderOutputSimpleMiddleware:class ProcessSpiderOutputAsyncGenMiddleware:async def process_spider_output(self, response, result, spider):- async for r in as_async_generator(result):+ async for r in result:yield rclass ProcessSpiderOutputUniversalMiddleware:def process_spider_output(self, response, result, spider):- @_process_iterable_universal- async def process(result):- async for r in result:- yield r- return process(result)+ for r in result:+ yield r++ async def process_spider_output_async(self, response, result, spider):+ async for r in result:+ yield r+++class ProcessSpiderExceptionSimpleIterableMiddleware:+ def process_spider_exception(self, response, exception, spider):+ yield {'foo': 1}+ yield {'foo': 2}+ yield {'foo': 3}+++class ProcessSpiderExceptionAsyncIterableMiddleware:+ async def process_spider_exception(self, response, exception, spider):+ yield {'foo': 1}+ d = defer.Deferred()+ from twisted.internet import reactor+ reactor.callLater(0, d.callback, None)+ await maybe_deferred_to_future(d)+ yield {'foo': 2}+ yield {'foo': 3}class ProcessSpiderOutputSimple(BaseAsyncSpiderMiddlewareTestCase):@@ -183,18 +211,19 @@ class ProcessSpiderOutputSimple(BaseAsyncSpiderMiddlewareTestCase):return self._test_simple_base(self.MW_SIMPLE)def test_asyncgen(self):- """ Asyncgen mw """+ """ Asyncgen mw; upgrade """return self._test_asyncgen_base(self.MW_ASYNCGEN)def test_simple_asyncgen(self):- """ Simple mw -> asyncgen mw """+ """ Simple mw -> asyncgen mw; upgrade """return self._test_asyncgen_base(self.MW_ASYNCGEN,self.MW_SIMPLE)def test_asyncgen_simple(self):- """ Asyncgen mw -> simple mw; cannot work """- return self._test_asyncgen_fail(self.MW_SIMPLE,- self.MW_ASYNCGEN)+ """ Asyncgen mw -> simple mw; upgrade then downgrade """+ return self._test_simple_base(self.MW_SIMPLE,+ self.MW_ASYNCGEN,+ downgrade=True)def test_universal(self):""" Universal mw """@@ -211,12 +240,12 @@ class ProcessSpiderOutputSimple(BaseAsyncSpiderMiddlewareTestCase):self.MW_SIMPLE)def test_universal_asyncgen(self):- """ Universal mw -> asyncgen mw """+ """ Universal mw -> asyncgen mw; upgrade """return self._test_asyncgen_base(self.MW_ASYNCGEN,self.MW_UNIVERSAL)def test_asyncgen_universal(self):- """ Asyncgen mw -> universal mw """+ """ Asyncgen mw -> universal mw; upgrade """return self._test_asyncgen_base(self.MW_UNIVERSAL,self.MW_ASYNCGEN)@@ -229,27 +258,31 @@ class ProcessSpiderOutputAsyncGen(ProcessSpiderOutputSimple):yield itemdef test_simple(self):- """ Simple mw; cannot work """- return self._test_asyncgen_fail(self.MW_SIMPLE)+ """ Simple mw; downgrade """+ return self._test_simple_base(self.MW_SIMPLE,+ downgrade=True)def test_simple_asyncgen(self):- """ Simple mw -> asyncgen mw; cannot work """- return self._test_asyncgen_fail(self.MW_ASYNCGEN,- self.MW_SIMPLE)+ """ Simple mw -> asyncgen mw; downgrade then upgrade """+ return self._test_asyncgen_base(self.MW_ASYNCGEN,+ self.MW_SIMPLE,+ downgrade=True)def test_universal(self):""" Universal mw """return self._test_asyncgen_base(self.MW_UNIVERSAL)def test_universal_simple(self):- """ Universal mw -> simple mw; cannot work """- return self._test_asyncgen_fail(self.MW_SIMPLE,- self.MW_UNIVERSAL)+ """ Universal mw -> simple mw; downgrade """+ return self._test_simple_base(self.MW_SIMPLE,+ self.MW_UNIVERSAL,+ downgrade=True)def test_simple_universal(self):- """ Simple mw -> universal mw; cannot work """- return self._test_asyncgen_fail(self.MW_UNIVERSAL,- self.MW_SIMPLE)+ """ Simple mw -> universal mw; downgrade """+ return self._test_simple_base(self.MW_UNIVERSAL,+ self.MW_SIMPLE,+ downgrade=True)class ProcessStartRequestsSimpleMiddleware:@@ -269,13 +302,198 @@ class ProcessStartRequestsSimple(BaseAsyncSpiderMiddlewareTestCase):yield Request(f'https://example.com/{i}', dont_filter=True)@defer.inlineCallbacks- def _get_middleware_result(self, *mw_classes):- for mw_cls in mw_classes:- self.mwman._add_middleware(mw_cls())+ def _get_middleware_result(self, *mw_classes, start_index: Optional[int] = None):+ setting = self._construct_mw_setting(*mw_classes, start_index=start_index)+ self.crawler = get_crawler(Spider, {'SPIDER_MIDDLEWARES_BASE': {}, 'SPIDER_MIDDLEWARES': setting})+ self.spider = self.crawler._create_spider('foo')+ self.mwman = SpiderMiddlewareManager.from_crawler(self.crawler)start_requests = iter(self._start_requests())results = yield self.mwman.process_start_requests(start_requests, self.spider)return resultsdef test_simple(self):""" Simple mw """- self._test_simple_base(self.MW_SIMPLE)+ return self._test_simple_base(self.MW_SIMPLE)+++class UniversalMiddlewareNoSync:+ async def process_spider_output_async(self, response, result, spider):+ yield+++class UniversalMiddlewareBothSync:+ def process_spider_output(self, response, result, spider):+ yield++ def process_spider_output_async(self, response, result, spider):+ yield+++class UniversalMiddlewareBothAsync:+ async def process_spider_output(self, response, result, spider):+ yield++ async def process_spider_output_async(self, response, result, spider):+ yield+++class UniversalMiddlewareManagerTest(TestCase):+ def setUp(self):+ self.mwman = SpiderMiddlewareManager()++ def test_simple_mw(self):+ mw = ProcessSpiderOutputSimpleMiddleware+ self.mwman._add_middleware(mw)+ self.assertEqual(self.mwman.methods['process_spider_output'][0], mw.process_spider_output)++ def test_async_mw(self):+ mw = ProcessSpiderOutputAsyncGenMiddleware+ self.mwman._add_middleware(mw)+ self.assertEqual(self.mwman.methods['process_spider_output'][0], mw.process_spider_output)++ def test_universal_mw(self):+ mw = ProcessSpiderOutputUniversalMiddleware+ self.mwman._add_middleware(mw)+ self.assertEqual(self.mwman.methods['process_spider_output'][0],+ (mw.process_spider_output, mw.process_spider_output_async))++ def test_universal_mw_no_sync(self):+ with LogCapture() as log:+ self.mwman._add_middleware(UniversalMiddlewareNoSync)+ self.assertIn("UniversalMiddlewareNoSync has process_spider_output_async"+ " without process_spider_output", str(log))+ self.assertEqual(self.mwman.methods['process_spider_output'][0], None)++ def test_universal_mw_both_sync(self):+ mw = UniversalMiddlewareBothSync+ with LogCapture() as log:+ self.mwman._add_middleware(mw)+ self.assertIn("UniversalMiddlewareBothSync.process_spider_output_async "+ "is not an async generator function", str(log))+ self.assertEqual(self.mwman.methods['process_spider_output'][0], mw.process_spider_output)++ def test_universal_mw_both_async(self):+ with LogCapture() as log:+ self.mwman._add_middleware(UniversalMiddlewareBothAsync)+ self.assertIn("UniversalMiddlewareBothAsync.process_spider_output "+ "is an async generator function while process_spider_output_async exists",+ str(log))+ self.assertEqual(self.mwman.methods['process_spider_output'][0], None)+++class BuiltinMiddlewareSimpleTest(BaseAsyncSpiderMiddlewareTestCase):+ ITEM_TYPE = dict+ MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware+ MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware+ MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware++ def _scrape_func(self, *args, **kwargs):+ yield {'foo': 1}+ yield {'foo': 2}+ yield {'foo': 3}++ @defer.inlineCallbacks+ def _get_middleware_result(self, *mw_classes, start_index: Optional[int] = None):+ setting = self._construct_mw_setting(*mw_classes, start_index=start_index)+ self.crawler = get_crawler(Spider, {'SPIDER_MIDDLEWARES': setting})+ self.spider = self.crawler._create_spider('foo')+ self.mwman = SpiderMiddlewareManager.from_crawler(self.crawler)+ result = yield self.mwman.scrape_response(self._scrape_func, self.response, self.request, self.spider)+ return result++ def test_just_builtin(self):+ return self._test_simple_base()++ def test_builtin_simple(self):+ return self._test_simple_base(self.MW_SIMPLE, start_index=1000)++ def test_builtin_async(self):+ """ Upgrade """+ return self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)++ def test_builtin_universal(self):+ return self._test_simple_base(self.MW_UNIVERSAL, start_index=1000)++ def test_simple_builtin(self):+ return self._test_simple_base(self.MW_SIMPLE)++ def test_async_builtin(self):+ """ Upgrade """+ return self._test_asyncgen_base(self.MW_ASYNCGEN)++ def test_universal_builtin(self):+ return self._test_simple_base(self.MW_UNIVERSAL)+++class BuiltinMiddlewareAsyncGenTest(BuiltinMiddlewareSimpleTest):+ async def _scrape_func(self, *args, **kwargs):+ for item in super()._scrape_func():+ yield item++ def test_just_builtin(self):+ return self._test_asyncgen_base()++ def test_builtin_simple(self):+ """ Downgrade """+ return self._test_simple_base(self.MW_SIMPLE, downgrade=True, start_index=1000)++ def test_builtin_async(self):+ return self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)++ def test_builtin_universal(self):+ return self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)++ def test_simple_builtin(self):+ """ Downgrade """+ return self._test_simple_base(self.MW_SIMPLE, downgrade=True)++ def test_async_builtin(self):+ return self._test_asyncgen_base(self.MW_ASYNCGEN)++ def test_universal_builtin(self):+ return self._test_asyncgen_base(self.MW_UNIVERSAL)+++class ProcessSpiderExceptionTest(BaseAsyncSpiderMiddlewareTestCase):+ ITEM_TYPE = dict+ MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware+ MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware+ MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware+ MW_EXC_SIMPLE = ProcessSpiderExceptionSimpleIterableMiddleware+ MW_EXC_ASYNCGEN = ProcessSpiderExceptionAsyncIterableMiddleware++ def _scrape_func(self, *args, **kwargs):+ 1 / 0++ @defer.inlineCallbacks+ def _test_asyncgen_nodowngrade(self, *mw_classes):+ with self.assertRaisesRegex(_InvalidOutput, "Async iterable returned from .+ cannot be downgraded"):+ yield self._get_middleware_result(*mw_classes)++ def test_exc_simple(self):+ """ Simple exc mw """+ return self._test_simple_base(self.MW_EXC_SIMPLE)++ def test_exc_async(self):+ """ Async exc mw """+ return self._test_asyncgen_base(self.MW_EXC_ASYNCGEN)++ def test_exc_simple_simple(self):+ """ Simple exc mw -> simple output mw """+ return self._test_simple_base(self.MW_SIMPLE,+ self.MW_EXC_SIMPLE)++ def test_exc_async_async(self):+ """ Async exc mw -> async output mw """+ return self._test_asyncgen_base(self.MW_ASYNCGEN,+ self.MW_EXC_ASYNCGEN)++ def test_exc_simple_async(self):+ """ Simple exc mw -> async output mw; upgrade """+ return self._test_asyncgen_base(self.MW_ASYNCGEN,+ self.MW_EXC_SIMPLE)++ def test_exc_async_simple(self):+ """ Async exc mw -> simple output mw; cannot work as downgrading is not supported """+ return self._test_asyncgen_nodowngrade(self.MW_SIMPLE,+ self.MW_EXC_ASYNCGEN)diff --git a/tests/test_spidermiddleware_output_chain.py b/tests/test_spidermiddleware_output_chain.pyindex 088c14ca8..dac246fb6 100644--- a/tests/test_spidermiddleware_output_chain.py+++ b/tests/test_spidermiddleware_output_chain.py@@ -28,6 +28,7 @@ class RecoveryMiddleware:class RecoverySpider(Spider):name = 'RecoverySpider'custom_settings = {+ 'SPIDER_MIDDLEWARES_BASE': {},'SPIDER_MIDDLEWARES': {RecoveryMiddleware: 10,},@@ -107,6 +108,13 @@ class GeneratorCallbackSpider(Spider):raise ImportError()+class AsyncGeneratorCallbackSpider(GeneratorCallbackSpider):+ async def parse(self, response):+ yield {'test': 1}+ yield {'test': 2}+ raise ImportError()++# ================================================================================# (2.1) exceptions from a spider callback (generator, middleware right after callback)class GeneratorCallbackSpiderMiddlewareRightAfterSpider(GeneratorCallbackSpider):@@ -360,6 +368,15 @@ class TestSpiderMiddleware(TestCase):self.assertIn("Middleware: ImportError exception caught", str(log2))self.assertIn("'item_scraped_count': 2", str(log2))+ @defer.inlineCallbacks+ def test_async_generator_callback(self):+ """+ Same as test_generator_callback but with an async callback.+ """+ log2 = yield self.crawl_log(AsyncGeneratorCallbackSpider)+ self.assertIn("Middleware: ImportError exception caught", str(log2))+ self.assertIn("'item_scraped_count': 2", str(log2))+@defer.inlineCallbacksdef test_generator_callback_right_after_callback(self):"""diff --git a/tests/test_utils_asyncgen.py b/tests/test_utils_asyncgen.pyindex 7abe17c22..9ae66c57c 100644--- a/tests/test_utils_asyncgen.py+++ b/tests/test_utils_asyncgen.py@@ -1,7 +1,6 @@-from twisted.internet.defer import Deferredfrom twisted.trial import unittest-from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen, _process_iterable_universal+from scrapy.utils.asyncgen import as_async_generator, collect_asyncgenfrom scrapy.utils.defer import deferred_f_from_coro_f@@ -19,52 +18,3 @@ class AsyncgenUtilsTest(unittest.TestCase):ag = as_async_generator(range(42))results = await collect_asyncgen(ag)self.assertEqual(results, list(range(42)))---@_process_iterable_universal-async def process_iterable(iterable):- async for i in iterable:- yield i * 2---@_process_iterable_universal-async def process_iterable_awaiting(iterable):- async for i in iterable:- yield i * 2- d = Deferred()- from twisted.internet import reactor- reactor.callLater(0, d.callback, 42)- await d---class ProcessIterableUniversalTest(unittest.TestCase):-- def test_normal(self):- iterable = iter([1, 2, 3])- results = list(process_iterable(iterable))- self.assertEqual(results, [2, 4, 6])-- @deferred_f_from_coro_f- async def test_async(self):- iterable = as_async_generator([1, 2, 3])- results = await collect_asyncgen(process_iterable(iterable))- self.assertEqual(results, [2, 4, 6])-- @deferred_f_from_coro_f- async def test_blocking(self):- iterable = [1, 2, 3]- with self.assertRaisesRegex(RuntimeError, "Synchronously-called function"):- list(process_iterable_awaiting(iterable))-- def test_invalid_iterable(self):- with self.assertRaisesRegex(TypeError, "Wrong iterable type"):- process_iterable(None)-- @deferred_f_from_coro_f- async def test_invalid_process(self):- @_process_iterable_universal- def process_iterable_invalid(iterable):- pass-- with self.assertRaisesRegex(ValueError, "process_async returned wrong type"):- list(process_iterable_invalid([]))