Scrapy, a fast high-level web crawling & scraping framework for Python.
Pause request processing based on the sum of body sizes of all active responses (#6430)
Adrián Chavesffe67f836f2b
diff --git a/docs/news/2.x.rst b/docs/news/2.x.rstindex dc0663207..6164b0cbc 100644--- a/docs/news/2.x.rst+++ b/docs/news/2.x.rst@@ -4045,9 +4045,9 @@ New features* A new ``DNS_RESOLVER`` setting allows enabling IPv6 support(:gh:`1031`, :gh:`4227`)-* A new :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` setting allows configuring- the existing soft limit that pauses request downloads when the total- response data being processed is too high (:gh:`1410`, :gh:`3551`)+* A new ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` setting allows configuring the+ existing soft limit that pauses request downloads when the total response+ data being processed is too high (:gh:`1410`, :gh:`3551`)* A new :setting:`TWISTED_REACTOR` setting allows customizing the:mod:`~twisted.internet.reactor` that Scrapy uses, allowing todiff --git a/docs/topics/api.rst b/docs/topics/api.rstindex 84befb2be..56c4f2819 100644--- a/docs/topics/api.rst+++ b/docs/topics/api.rst@@ -208,7 +208,8 @@ Engine API==========.. autoclass:: scrapy.core.engine.ExecutionEngine()- :members: close_spider_async, needs_backout, running, scheduler, state+ :members: close_spider_async, needs_backout, response_active_size, running,+ scheduler, state.. autoclass:: scrapy.core.engine.EngineState:members:diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rstindex c1e88a32c..f1c0a019b 100644--- a/docs/topics/optimize.rst+++ b/docs/topics/optimize.rst@@ -41,8 +41,7 @@ what every part of the engine is doing at a given moment:len(engine.downloader.active) : 16len(engine.scheduler.mqs) : 92len(engine.scraper.slot.active) : 0- engine.scraper.slot.active_size : 0- engine.scraper.slot.needs_backout() : False+ engine.response_active_size : 1310720A coding agent using the :ref:`Scrapy MCP server <using-mcp-server>` can alsoretrieve these values directly.@@ -64,10 +63,13 @@ Take a few readings at different points of the crawl:page at a time cannot use more concurrency than it creates. See:ref:`optimize-requests`.-- ``needs_backout()`` is ``True``, or ``active_size`` approaches- :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`: responses arrive faster than your- callbacks and :ref:`item pipelines <topics-item-pipeline>` handle them. The- bottleneck is your own code.+- :attr:`engine.response_active_size+ <scrapy.core.engine.ExecutionEngine.response_active_size>` approaches+ :setting:`RESPONSE_MAX_ACTIVE_SIZE`, or the+ :stat:`request_backout_seconds/response_max_active_size` stat grows (see+ :ref:`crawl-bottlenecks`): responses arrive faster than your callbacks and+ :ref:`item pipelines <topics-item-pipeline>` handle them, or your code+ keeps responses in memory. The bottleneck is your own code.- ``len(engine.scheduler.mqs)`` grows without settling: the crawl discoversrequests faster than it downloads them. This is what makes long crawls run@@ -252,7 +254,7 @@ Lowering resource usageLowering memory usage----------------------- Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.+- Lower :setting:`RESPONSE_MAX_ACTIVE_SIZE`.- Lower :setting:`DOWNLOAD_MAXSIZE`, which allows a single response to take upto 1 GiB of memory by default, multiplied by your concurrency. Set@@ -373,7 +375,7 @@ For broad crawls, consider these adjustments:(e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memoryallow.- - Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing+ - Increase :setting:`RESPONSE_MAX_ACTIVE_SIZE` when increasing:setting:`CONCURRENT_REQUESTS` stops making a difference.- .. _broad-crawls-bfo:diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 0503cefab..bbfba806f 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -534,6 +534,31 @@ run:Because :setting:`SPIDER_MODULES` is a list setting, you can include multiplemodules by separating them with commas.+.. _crawl-bottlenecks:++Identifying crawl bottlenecks+=============================++The :stat:`request_backout_seconds/concurrency`,+:stat:`request_backout_seconds/response_max_active_size` and+:stat:`request_backout_seconds/total` stats show how long request scheduling+was paused during a crawl, and why.++For example, after a crawl you might see::++ 2025-01-01 00:00:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:+ {'request_backout_seconds/concurrency': 12.5,+ 'request_backout_seconds/response_max_active_size': 45.2,+ 'request_backout_seconds/total': 57.7,+ ...}++In this case, the spider spent about 45 seconds paused due to large responses+in memory. You could:++- Increase :setting:`RESPONSE_MAX_ACTIVE_SIZE` if your machine has enough RAM.+- Check that your code doesn't hold strong references to+ :class:`~scrapy.http.Response` objects longer than necessary.+.. _bans:Avoiding getting banneddiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex c0d25ad94..a5316ca38 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -2079,6 +2079,53 @@ Adjust redirect request priority relative to original request:- **a positive priority adjust (default) means higher priority.**- a negative priority adjust means lower priority.+.. setting:: RESPONSE_MAX_ACTIVE_SIZE++RESPONSE_MAX_ACTIVE_SIZE+------------------------++.. versionadded:: VERSION++Default: ``5_000_000``++Soft limit (in bytes) on the total size of responses being kept in memory.++This counts both the size of response bodies that have passed through+:ref:`downloader middlewares <topics-downloader-middleware>` and remain in+memory, and the :setting:`rough size <RESPONSE_ROUGH_SIZE>` of requests+currently being downloaded.++While the total is above this value, Scrapy pauses sending new requests to the+downloader. A higher value improves crawl speed at the cost of memory usage.+``0`` disables the limit.++Scrapy logs an info-level message the first time the limit is reached. To see+how long request processing was paused because of it over a whole crawl, check+the :stat:`request_backout_seconds/response_max_active_size` stat.++.. caution::++ Responses that your code keeps a strong reference to, e.g. in the+ :attr:`.Request.meta` of a scheduled request or in a component attribute,+ count toward this limit until that reference is gone, so accumulating them+ can pause a crawl indefinitely. So do responses in a reference cycle,+ until Python's garbage collector frees them. To find out whether your+ crawl keeps responses alive, see :ref:`topics-leaks`.++.. setting:: RESPONSE_ROUGH_SIZE++RESPONSE_ROUGH_SIZE+-------------------++.. versionadded:: VERSION++Default: ``None``++Estimated size (in bytes) to count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`+for each request being downloaded, whose actual response size is not known yet.++``None`` lets Scrapy choose a value. ``0`` counts only responses.+.. setting:: ROBOTSTXT_OBEYROBOTSTXT_OBEY@@ -2242,19 +2289,6 @@ For available choices, see :setting:`SCHEDULER_MEMORY_QUEUE`.:start-after: queue-common-starts:end-before: queue-common-ends--.. setting:: SCRAPER_SLOT_MAX_ACTIVE_SIZE--SCRAPER_SLOT_MAX_ACTIVE_SIZE-------------------------------Default: ``5_000_000``--Soft limit (in bytes) for response data being processed.--While the sum of the sizes of all responses being processed is above this value,-Scrapy does not process new requests.-.. setting:: SPIDER_CONTRACTSSPIDER_CONTRACTSdiff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 8eb628c2d..cc1b12d9f 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -173,8 +173,8 @@ scheduler_empty.. warning:: Only wait for this signal from:meth:`~scrapy.Spider.start`. While no request can be sent, e.g. while- the responses being parsed exceed- :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`, the engine does not ask the+ the responses in memory exceed+ :setting:`RESPONSE_MAX_ACTIVE_SIZE`, the engine does not ask thescheduler for requests, and hence this signal is not sent. So waitingfor it from a :ref:`callback <callbacks>` can hang the crawl,because the response being parsed is itself one of the responses thatdiff --git a/docs/topics/stats.rst b/docs/topics/stats.rstindex 57b506a5e..1dd64297b 100644--- a/docs/topics/stats.rst+++ b/docs/topics/stats.rst@@ -610,6 +610,24 @@ one per actual value of the placeholder.Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.+.. stat:: request_backout_seconds/concurrency++``request_backout_seconds/concurrency``+ Seconds during which sending requests to the downloader was paused because+ :setting:`CONCURRENT_REQUESTS` was reached.++.. stat:: request_backout_seconds/response_max_active_size++``request_backout_seconds/response_max_active_size``+ Seconds during which sending requests to the downloader was paused because+ :setting:`RESPONSE_MAX_ACTIVE_SIZE` was reached.++.. stat:: request_backout_seconds/total++``request_backout_seconds/total``+ Seconds during which sending requests to the downloader was paused, for+ any of the reasons above.+.. stat:: request_depth_count/{depth}``request_depth_count/{depth}``diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rstindex f23bdf37d..b4770219e 100644--- a/docs/topics/telnetconsole.rst+++ b/docs/topics/telnetconsole.rst@@ -125,9 +125,8 @@ engine status::len(engine.scheduler.mqs) : 92len(engine.scraper.slot.queue) : 0len(engine.scraper.slot.active) : 0- engine.scraper.slot.active_size : 0+ engine.response_active_size : 1310720engine.scraper.slot.itemproc_size : 0- engine.scraper.slot.needs_backout() : FalsePause, resume and stop the Scrapy enginediff --git a/pyproject.toml b/pyproject.tomlindex 064ce6a05..89355c49c 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -8,7 +8,7 @@ dynamic = ["version"]description = "A high-level Web Crawling and Web Scraping framework"dependencies = ["PyDispatcher>=2.0.5",- "Twisted>=21.7.0",+ "Twisted>=23.8.0","aiodns>=3.3.0","aioftp>=0.26.1","aiohttp>=3.13.3",@@ -79,7 +79,7 @@ ptpython = ["ptpython>=3.0.23"]robotparser = ["robotexclusionrulesparser>=1.6.2"]s3 = ["aioboto3>=9.0.0", "boto3>=1.35.2"]socks = ["python-socks[asyncio]>=2.6.0"]-twisted-http2 = ["Twisted[http2]>=21.7.0"]+twisted-http2 = ["Twisted[http2]>=23.8.0"]uvloop = ["uvloop>=0.17.0; platform_system != 'Windows'",]diff --git a/scrapy/core/downloader/__init__.py b/scrapy/core/downloader/__init__.pyindex f851ed745..babb9a010 100644--- a/scrapy/core/downloader/__init__.py+++ b/scrapy/core/downloader/__init__.py@@ -5,7 +5,8 @@ import randomimport warningsfrom collections import dequefrom dataclasses import dataclass, field-from time import monotonic, time+from logging import getLogger+from time import monotonic, perf_counter, timefrom typing import TYPE_CHECKING, Anyfrom twisted.internet.defer import Deferred, DeferredList, inlineCallbacks@@ -42,6 +43,8 @@ if TYPE_CHECKING:from scrapy.settings import BaseSettingsfrom scrapy.signalmanager import SignalManager+logger = getLogger(__name__)+# Request.meta key holding the delay of a request and the time before which it# must not be sent because of it. The deadline is a wall-clock time, so that it@@ -233,6 +236,11 @@ class Downloader:self.per_slot_settings: dict[str, dict[str, Any]] = self.settings.getdict("DOWNLOAD_SLOTS")+ self._stats = crawler.stats+ # (reason, start_time): current backout reason and when it began, or+ # (None, None) if not backing out.+ self._last_backout: tuple[str | None, float | None] = (None, None)+ self._max_active_size_warned = False@propertydef randomize_delay(self) -> bool:@@ -250,6 +258,7 @@ class Downloader:self, request: Request, spider: Spider | None = None) -> Generator[Deferred[Any], Any, Response | Request]:self.active.add(request)+ self.middleware._count_rough_size(request)try:result: Response | Request = yield (deferred_from_coro(@@ -259,10 +268,50 @@ class Downloader:return resultfinally:self.active.remove(request)+ self.middleware._discount_rough_size(request)++ def _record_backout(self, reason: str | None) -> None:+ last_reason, last_reason_start_time = self._last_backout+ if last_reason == reason:+ return+ current_time = perf_counter()+ if last_reason is not None and self._stats is not None:+ assert last_reason_start_time is not None+ last_reason_seconds = current_time - last_reason_start_time+ self._stats.inc_value("request_backout_seconds/total", last_reason_seconds)+ self._stats.inc_value(+ f"request_backout_seconds/{last_reason}", last_reason_seconds+ )+ self._last_backout = (reason, current_time)def needs_backout(self) -> bool:# A total concurrency of 0 means no limit.- return 0 < self.total_concurrency <= len(self.active)+ if 0 < self.total_concurrency <= len(self.active):+ self._record_backout("concurrency")+ return True+ if self._exceeds_max_active_size():+ self._record_backout("response_max_active_size")+ if not self._max_active_size_warned:+ self._max_active_size_warned = True+ logger.info(+ f"Pausing request processing: the active response size "+ f"({self.middleware._total_active_size} B) has reached "+ f"RESPONSE_MAX_ACTIVE_SIZE "+ f"({self.middleware._max_active_size} B). See "+ f"https://docs.scrapy.org/en/latest/topics/settings.html#response-max-active-size "+ f"and the request_backout_seconds/response_max_active_size "+ f"stat. This message is only logged once."+ )+ return True+ self._record_backout(None)+ return False++ def _exceeds_max_active_size(self) -> bool:+ max_active_size = self.middleware._max_active_size+ return (+ bool(max_active_size)+ and self.middleware._total_active_size >= max_active_size+ )@_warn_spider_argdef _get_slot(@@ -489,6 +538,7 @@ class Downloader:self._stop_slot_gc()for slot in self.slots.values():slot.close()+ self._record_backout(None)async def _close_async(self) -> None:"""Close the downloader and its download handlers."""diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.pyindex 14e11516e..8c316f6fd 100644--- a/scrapy/core/downloader/middleware.py+++ b/scrapy/core/downloader/middleware.py@@ -7,12 +7,13 @@ See documentation in docs/topics/downloader-middleware.rstfrom __future__ import annotationsimport warnings-from functools import wraps+from functools import partial, wrapsfrom itertools import islicefrom typing import TYPE_CHECKING, Any+from weakref import WeakSet, finalizefrom scrapy.exceptions import CloseSpider, ScrapyDeprecationWarning, _InvalidOutput-from scrapy.http import Request, Response+from scrapy.http import BodyFileResponse, Request, Responsefrom scrapy.middleware import MiddlewareManagerfrom scrapy.utils.conf import build_component_listfrom scrapy.utils.defer import (@@ -33,15 +34,62 @@ if TYPE_CHECKING:from scrapy.settings import BaseSettings+def _get_max_active_size(settings: BaseSettings) -> int:+ deprecated_priority = settings.getpriority("SCRAPER_SLOT_MAX_ACTIVE_SIZE")+ priority = settings.getpriority("RESPONSE_MAX_ACTIVE_SIZE")+ assert deprecated_priority is not None+ assert priority is not None+ if deprecated_priority <= 0:+ return settings.getint("RESPONSE_MAX_ACTIVE_SIZE")+ if priority >= deprecated_priority:+ warnings.warn(+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated and is "+ "being ignored because RESPONSE_MAX_ACTIVE_SIZE is set with an "+ "equal or higher priority. Remove SCRAPER_SLOT_MAX_ACTIVE_SIZE "+ "from your settings.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ return settings.getint("RESPONSE_MAX_ACTIVE_SIZE")+ warnings.warn(+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated, use "+ "RESPONSE_MAX_ACTIVE_SIZE instead.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ return settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE")+++def _get_response_rough_size(settings: BaseSettings, max_active_size: int) -> int:+ if settings.get("RESPONSE_ROUGH_SIZE") is not None:+ return settings.getint("RESPONSE_ROUGH_SIZE")+ concurrency = settings.getint("CONCURRENT_REQUESTS")+ if not concurrency:+ # Unlimited concurrency: there is no bound on the number of in-flight+ # requests to spread a share of the limit over.+ return 0+ # Requests being downloaded, whose response size is unknown, may take up to+ # a quarter of the limit; the rest is for responses already in memory.+ return max_active_size // (4 * concurrency)++class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstract-methodcomponent_name = "downloader middleware"def __init__(self, *middlewares: Any, crawler: Crawler | None = None) -> None:super().__init__(*middlewares, crawler=crawler)- self._response_exceptions = (- crawler.settings.getbool("DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS")- if crawler is not None- else False+ assert crawler is not None+ settings = crawler.settings+ self._max_active_size: int = _get_max_active_size(settings)+ self._response_rough_size: int = _get_response_rough_size(+ settings, self._max_active_size+ )+ self._response_active_size = 0+ self._tracked_responses: WeakSet[Response] = WeakSet()+ self._rough_active_size = 0+ self._rough_sizes: dict[Request, int] = {}+ self._response_exceptions = settings.getbool(+ "DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS")@classmethod@@ -63,6 +111,39 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstracif process_exception is not None:self._check_mw_method_spider_arg(process_exception)+ @property+ def _total_active_size(self) -> int:+ return self._response_active_size + self._rough_active_size++ def _count_rough_size(self, request: Request) -> None:+ # Counting the same request twice, which concurrent downloads of the+ # same request object would do, would leak its rough size, since only+ # one discount call can find it.+ if request in self._rough_sizes:+ return+ self._rough_sizes[request] = self._response_rough_size+ self._rough_active_size += self._response_rough_size++ def _discount_rough_size(self, request: Request) -> None:+ self._rough_active_size -= self._rough_sizes.pop(request, 0)++ def _count_response_size(self, response: Response, request: Request) -> None:+ # The actual response size replaces the rough size estimate of its+ # request.+ self._discount_rough_size(request)+ if (+ isinstance(response, BodyFileResponse)+ or response in self._tracked_responses+ ):+ return+ self._tracked_responses.add(response)+ size = len(response.body)+ self._response_active_size += size+ finalize(response, partial(self._discount_response_size, size))++ def _discount_response_size(self, size: int) -> None:+ self._response_active_size -= size+def download(self,download_func: Callable[[Request, Spider], Deferred[Response]],@@ -122,8 +203,13 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstracf"Request, got {response.__class__.__name__}")if response:+ if isinstance(response, Response):+ self._count_response_size(response, request)return response- return await download_func(request)+ response = await download_func(request)+ if response is not None:+ self._count_response_size(response, request)+ return responseasync def _process_response(self, response: Response | Request, request: Request, index: int = 0@@ -160,6 +246,8 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstracf"got {type(result)}")response = result+ if isinstance(response, Response):+ self._count_response_size(response, request)return responsedef _warn_response_exception(@@ -226,5 +314,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstracf"Request, got {type(response)}")if response:+ if isinstance(response, Response):+ self._count_response_size(response, request)return response, indexraise exceptiondiff --git a/scrapy/core/engine.py b/scrapy/core/engine.pyindex 790718ddd..ded2ba30e 100644--- a/scrapy/core/engine.py+++ b/scrapy/core/engine.py@@ -278,6 +278,14 @@ class ExecutionEngine:"""return self._state+ @property+ def response_active_size(self) -> int:+ """Size, in bytes, that counts toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`.++ .. versionadded:: VERSION+ """+ return self.downloader.middleware._total_active_size+@propertydef running(self) -> bool:"""Whether the engine is in the :attr:`EngineState.RUNNING` state.@@ -511,7 +519,6 @@ class ExecutionEngine:or not self._slotor bool(self._slot.closing)or self.downloader.needs_backout()- or self.scraper.slot.needs_backout())def _remove_request(self, _: Any, request: Request) -> None:diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.pyindex c1915436d..bdf3d5801 100644--- a/scrapy/core/scraper.py+++ b/scrapy/core/scraper.py@@ -20,7 +20,7 @@ from scrapy.exceptions import (IgnoreRequest,ScrapyDeprecationWarning,)-from scrapy.http import BodyFileResponse, Request, Response+from scrapy.http import Request, Responsefrom scrapy.http.request import _active_crawler_scopefrom scrapy.pipelines import ItemPipelineManagerfrom scrapy.utils.asyncgen import as_async_generator@@ -76,16 +76,13 @@ async def _aset_parent_id(yield o-class Slot:+class _Slot:"""Scraper slot (one per running spider)"""- MIN_RESPONSE_SIZE = 1024-- def __init__(self, max_active_size: int = 5000000):- self.max_active_size: int = max_active_size+ def __init__(self, crawler: Crawler) -> None:+ self._crawler = crawlerself.queue: deque[QueueTuple] = deque()self.active: set[Request] = set()- self.active_size: int = 0self.itemproc_size: int = 0self.closing: Deferred[Spider] | None = None@@ -95,10 +92,6 @@ class Slot:# this Deferred will be awaited in enqueue_scrape()deferred: Deferred[None] = Deferred()self.queue.append((result, request, deferred))- if isinstance(result, Response) and not isinstance(result, BodyFileResponse):- self.active_size += max(len(result.body), self.MIN_RESPONSE_SIZE)- else:- self.active_size += self.MIN_RESPONSE_SIZEreturn deferreddef next_response_request_deferred(self) -> QueueTuple:@@ -106,23 +99,48 @@ class Slot:self.active.add(request)return result, request, deferred- def finish_response(self, result: Response | Failure, request: Request) -> None:+ def finish_response(self, request: Request) -> None:self.active.remove(request)- if isinstance(result, Response) and not isinstance(result, BodyFileResponse):- self.active_size -= max(len(result.body), self.MIN_RESPONSE_SIZE)- else:- self.active_size -= self.MIN_RESPONSE_SIZEdef is_idle(self) -> bool:return not (self.queue or self.active or self.itemproc_size)+ @property+ def active_size(self) -> int:+ warnings.warn(+ "Scraper.slot.active_size is deprecated, use "+ "ExecutionEngine.response_active_size instead.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ assert self._crawler.engine is not None+ return self._crawler.engine.response_active_size++ @property+ def max_active_size(self) -> int:+ warnings.warn(+ "Scraper.slot.max_active_size is deprecated, use the "+ "RESPONSE_MAX_ACTIVE_SIZE setting instead.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ assert self._crawler.engine is not None+ return self._crawler.engine.downloader.middleware._max_active_size+def needs_backout(self) -> bool:- return self.active_size > self.max_active_size+ warnings.warn(+ "Scraper.slot.needs_backout() is deprecated, use "+ "ExecutionEngine.needs_backout() instead.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ assert self._crawler.engine is not None+ return self._crawler.engine.downloader._exceeds_max_active_size()class Scraper:def __init__(self, crawler: Crawler) -> None:- self.slot: Slot | None = None+ self.slot: _Slot | None = Noneself.spidermw: SpiderMiddlewareManager = build_from_crawler(SpiderMiddlewareManager, crawler)@@ -185,7 +203,7 @@ class Scraper:.. versionadded:: 2.14"""- self.slot = Slot(self.crawler.settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE"))+ self.slot = _Slot(self.crawler)if not self.crawler.spider:raise RuntimeError("Scraper.open_spider() called before Crawler.spider is set."@@ -250,7 +268,7 @@ class Scraper:extra={"spider": self.crawler.spider},)finally:- self.slot.finish_response(result, request)+ self.slot.finish_response(request)self._check_if_closing()self._scrape_next()@@ -602,3 +620,14 @@ class Scraper:finally:self.slot.itemproc_size -= 1self._check_if_closing()+++def __getattr__(name: str) -> Any:+ if name == "Slot":+ warnings.warn(+ "scrapy.core.scraper.Slot is deprecated.",+ ScrapyDeprecationWarning,+ stacklevel=2,+ )+ return _Slot+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.pyindex 69560bc67..0c162114b 100644--- a/scrapy/settings/default_settings.py+++ b/scrapy/settings/default_settings.py@@ -201,6 +201,8 @@ __all__ = ["REMOTE_CONTROL_TIMEOUT_MAX","REMOTE_CONTROL_TRACEBACK_MAX_BYTES","REQUEST_FINGERPRINTER_CLASS",+ "RESPONSE_MAX_ACTIVE_SIZE",+ "RESPONSE_ROUGH_SIZE","RETRY_ENABLED","RETRY_EXCEPTIONS","RETRY_GIVE_UP_LOG_LEVEL",@@ -605,7 +607,9 @@ SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue"SCHEDULER_START_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"SCHEDULER_START_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"-SCRAPER_SLOT_MAX_ACTIVE_SIZE = 5000000+SCRAPER_SLOT_MAX_ACTIVE_SIZE = 5_000_000+RESPONSE_MAX_ACTIVE_SIZE = 5_000_000+RESPONSE_ROUGH_SIZE = NoneSPIDER_CONTRACTS: dict[str, int] = {}SPIDER_CONTRACTS_BASE = {diff --git a/scrapy/statscollectors.py b/scrapy/statscollectors.pyindex f012109db..fdb2986de 100644--- a/scrapy/statscollectors.py+++ b/scrapy/statscollectors.py@@ -70,7 +70,11 @@ class StatsCollector(MutableMapping[str, Any]):self._stats = statsdef inc_value(- self, key: str, count: int = 1, start: int = 0, spider: Spider | None = None+ self,+ key: str,+ count: float = 1,+ start: float = 0,+ spider: Spider | None = None,) -> None:"""Increment the *key* stat by *count*, or set it to *start* if it isnot set."""@@ -176,7 +180,11 @@ class DummyStatsCollector(StatsCollector):passdef inc_value(- self, key: str, count: int = 1, start: int = 0, spider: Spider | None = None+ self,+ key: str,+ count: float = 1,+ start: float = 0,+ spider: Spider | None = None,) -> None:passdiff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.pyindex 2b1d68709..603667b4d 100644--- a/scrapy/utils/defer.py+++ b/scrapy/utils/defer.py@@ -239,6 +239,10 @@ class _ParallelTasks:# Only a running task can start another one, so once none is left no# more work can come.if not self._running:+ # The work usually references self.start, so keeping it would+ # leave a reference cycle that keeps the work arguments, e.g. a+ # response, alive until the next garbage collection.+ del self._workself.finished.callback(None)diff --git a/scrapy/utils/engine.py b/scrapy/utils/engine.pyindex 2261bfe58..3ca3ec12e 100644--- a/scrapy/utils/engine.py+++ b/scrapy/utils/engine.py@@ -24,9 +24,8 @@ def get_engine_status(engine: ExecutionEngine) -> list[tuple[str, object]]:"len(engine.scheduler.mqs)","len(engine.scraper.slot.queue)","len(engine.scraper.slot.active)",- "engine.scraper.slot.active_size",+ "engine.response_active_size","engine.scraper.slot.itemproc_size",- "engine.scraper.slot.needs_backout()",]checks: list[tuple[str, object]] = []diff --git a/tests/test_core_scraper.py b/tests/test_core_scraper.pyindex c4732ff3f..f5b5c7dc2 100644--- a/tests/test_core_scraper.py+++ b/tests/test_core_scraper.py@@ -6,8 +6,9 @@ from typing import TYPE_CHECKING, Any, castimport pytestfrom scrapy import Request, Spider-from scrapy.core.scraper import Scraper-from scrapy.exceptions import IgnoreRequest+from scrapy.core import scraper as scraper_module+from scrapy.core.scraper import Scraper, _Slot+from scrapy.exceptions import IgnoreRequest, ScrapyDeprecationWarningfrom scrapy.http import Responsefrom scrapy.utils.defer import maybe_deferred_to_futurefrom scrapy.utils.spider import DefaultSpider@@ -137,3 +138,45 @@ async def test_none_in_callback_output() -> None:)await crawler.crawl_async()assert crawler.stats.get_value("item_scraped_count") == 2+++@coroutine_test+async def test_slot_deprecated_attributes() -> None:+ values: dict[str, Any] = {}++ class SizeSpider(Spider):+ name = "size"++ async def start(self) -> AsyncIterator[Any]:+ yield Request("data:,a")++ def parse(self, response: Response) -> None:+ engine = self.crawler.engine+ assert engine is not None+ slot = engine.scraper.slot+ assert slot is not None+ values["response_active_size"] = engine.response_active_size+ for name in ("active_size", "max_active_size"):+ with pytest.warns(ScrapyDeprecationWarning, match=name):+ values[name] = getattr(slot, name)+ with pytest.warns(ScrapyDeprecationWarning, match="needs_backout"):+ values["needs_backout"] = slot.needs_backout()++ crawler = get_crawler(SizeSpider, {"RESPONSE_MAX_ACTIVE_SIZE": 1})+ await crawler.crawl_async()+ assert values == {+ "response_active_size": 1,+ "active_size": 1,+ "max_active_size": 1,+ "needs_backout": True,+ }+++def test_slot_deprecated() -> None:+ with pytest.warns(ScrapyDeprecationWarning, match="Slot is deprecated"):+ assert scraper_module.Slot is _Slot+++def test_missing_attribute() -> None:+ with pytest.raises(AttributeError):+ scraper_module.Foodiff --git a/tests/test_downloader.py b/tests/test_downloader.pynew file mode 100644index 000000000..ed123efef--- /dev/null+++ b/tests/test_downloader.py@@ -0,0 +1,658 @@+import asyncio+import gc+import logging+import warnings+from typing import Any++import pytest+from twisted.internet.defer import Deferred++from scrapy import Request, Spider+from scrapy.crawler import Crawler+from scrapy.exceptions import ScrapyDeprecationWarning+from scrapy.http import HtmlResponse, Response+from scrapy.utils.asyncio import call_later, sleep+from scrapy.utils.defer import maybe_deferred_to_future+from scrapy.utils.test import get_crawler+from tests.utils.decorators import coroutine_test+++def _count_backout_logs(caplog: pytest.LogCaptureFixture) -> int:+ return sum(+ 1+ for record in caplog.records+ if record.levelname == "INFO"+ and str(record.msg).startswith("Pausing request processing")+ )+++def _backout_stats(crawler: Crawler) -> dict[str, Any]:+ assert crawler.stats+ return {+ k: v+ for k, v in crawler.stats.get_stats().items()+ if k.startswith("request_backout_seconds/")+ }+++class OfflineSpider(Spider):+ name = "offline"+ start_urls = ["data:,"]++ def parse(self, response):+ pass+++def _assert_scraper_slot_deprecation(+ warning_messages: pytest.WarningsRecorder, *, ignored: bool = False+) -> None:+ """Assert that a crawl emitted exactly one Scrapy deprecation warning, the+ one about SCRAPER_SLOT_MAX_ACTIVE_SIZE.++ Only Scrapy deprecation warnings are counted: a crawl may emit unrelated+ warnings (e.g. a ResourceWarning for a socket garbage-collected while the+ recorder is active), and those must not make the assertion flaky.++ Pass ``ignored=True`` when RESPONSE_MAX_ACTIVE_SIZE is set with an equal or+ higher priority and therefore SCRAPER_SLOT_MAX_ACTIVE_SIZE is being+ ignored."""+ deprecations = [+ message+ for message in warning_messages+ if issubclass(message.category, ScrapyDeprecationWarning)+ ]+ assert len(deprecations) == 1+ if ignored:+ assert str(deprecations[0].message) == (+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated and is "+ "being ignored because RESPONSE_MAX_ACTIVE_SIZE is set with an "+ "equal or higher priority. Remove SCRAPER_SLOT_MAX_ACTIVE_SIZE "+ "from your settings."+ )+ else:+ assert str(deprecations[0].message) == (+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated, use "+ "RESPONSE_MAX_ACTIVE_SIZE instead."+ )+++def _raise(response: Response) -> None:+ raise ValueError(response.url)+++class gt:+ __hash__ = None # type: ignore[assignment]++ def __init__(self, value: float):+ self.value = value++ def __eq__(self, other: object) -> bool:+ return isinstance(other, (int, float)) and other > self.value++ def __repr__(self) -> str:+ return f">{self.value}"+++class TestResponseMaxActiveSize:+ @coroutine_test+ async def test_default(self):+ """A crawl without custom settings has its effective response max+ active size set to 5 000 000, and triggers no deprecation warning."""+ crawler = get_crawler(OfflineSpider)+ with warnings.catch_warnings():+ warnings.simplefilter("error", ScrapyDeprecationWarning)+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 5_000_000++ @coroutine_test+ async def test_custom(self):+ """Setting RESPONSE_MAX_ACTIVE_SIZE to a custom value changes the+ effective response max active size."""+ crawler = get_crawler(+ OfflineSpider, settings_dict={"RESPONSE_MAX_ACTIVE_SIZE": 0}+ )+ with warnings.catch_warnings():+ warnings.simplefilter("error", ScrapyDeprecationWarning)+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 0++ @coroutine_test+ async def test_deprecated_default(self):+ """Setting SCRAPER_SLOT_MAX_ACTIVE_SIZE triggers a deprecation warning,+ even if it is the default value."""+ crawler = get_crawler(+ OfflineSpider, settings_dict={"SCRAPER_SLOT_MAX_ACTIVE_SIZE": 5_000_000}+ )+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 5_000_000+ _assert_scraper_slot_deprecation(warning_messages)++ @coroutine_test+ async def test_deprecated_custom(self):+ """Setting SCRAPER_SLOT_MAX_ACTIVE_SIZE to a custom value triggers a+ deprecation warning, and changes the effective response max active+ size."""+ crawler = get_crawler(+ OfflineSpider, settings_dict={"SCRAPER_SLOT_MAX_ACTIVE_SIZE": 0}+ )+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 0+ _assert_scraper_slot_deprecation(warning_messages)++ @coroutine_test+ async def test_both(self):+ """Setting RESPONSE_MAX_ACTIVE_SIZE and SCRAPER_SLOT_MAX_ACTIVE_SIZE to+ different values with the same setting priority triggers a deprecation+ warning about SCRAPER_SLOT_MAX_ACTIVE_SIZE being ignored, and makes+ the value of RESPONSE_MAX_ACTIVE_SIZE the effective response max active+ size."""+ crawler = get_crawler(+ OfflineSpider,+ settings_dict={+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ "SCRAPER_SLOT_MAX_ACTIVE_SIZE": 2,+ },+ )+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 1+ _assert_scraper_slot_deprecation(warning_messages, ignored=True)++ @coroutine_test+ async def test_both_deprecated_priority(self):+ """Setting RESPONSE_MAX_ACTIVE_SIZE and SCRAPER_SLOT_MAX_ACTIVE_SIZE to+ different values and SCRAPER_SLOT_MAX_ACTIVE_SIZE with a higher+ priority triggers a deprecation warning about+ SCRAPER_SLOT_MAX_ACTIVE_SIZE but also makes the value of+ SCRAPER_SLOT_MAX_ACTIVE_SIZE the effective response max active size."""++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]++ @classmethod+ def update_settings(cls, settings):+ settings.set("RESPONSE_MAX_ACTIVE_SIZE", 1, priority=100)+ settings.set("SCRAPER_SLOT_MAX_ACTIVE_SIZE", 2, priority=101)++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._max_active_size == 2+ _assert_scraper_slot_deprecation(warning_messages)+++class TestResponseRoughSize:+ @pytest.fixture(autouse=True)+ def use_caplog(self, caplog):+ self.caplog = caplog++ @pytest.mark.parametrize(+ ("settings_dict", "expected"),+ [+ # A quarter of RESPONSE_MAX_ACTIVE_SIZE split among+ # CONCURRENT_REQUESTS requests.+ ({}, 78125),+ ({"CONCURRENT_REQUESTS": 100}, 12500),+ ({"RESPONSE_MAX_ACTIVE_SIZE": 8_000_000}, 125_000),+ # Unlimited concurrency leaves no number of requests to split it+ # among.+ ({"CONCURRENT_REQUESTS": 0}, 0),+ ({"RESPONSE_MAX_ACTIVE_SIZE": 0}, 0),+ ({"RESPONSE_ROUGH_SIZE": 1}, 1),+ ({"RESPONSE_ROUGH_SIZE": 0}, 0),+ ],+ )+ @coroutine_test+ async def test_value(self, settings_dict, expected):+ crawler = get_crawler(OfflineSpider, settings_dict=settings_dict)+ with warnings.catch_warnings():+ warnings.simplefilter("error", ScrapyDeprecationWarning)+ await maybe_deferred_to_future(crawler.crawl())+ assert crawler.engine+ assert crawler.engine.downloader.middleware._response_rough_size == expected++ @coroutine_test+ async def test_response_replaces_rough_size(self):+ """The rough size of a request stops counting as soon as the size of its+ response is known."""+ sizes: list[tuple[int, int]] = []++ class DownloaderMiddleware:+ def __init__(self, crawler):+ self.crawler = crawler++ @classmethod+ def from_crawler(cls, crawler):+ return cls(crawler)++ def process_response(self, request, response):+ middleware = self.crawler.engine.downloader.middleware+ sizes.append(+ (middleware._rough_active_size, middleware._response_active_size)+ )+ return response++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,a"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ await maybe_deferred_to_future(crawler.crawl())++ assert sizes == [(0, 1)]+ assert crawler.engine+ assert crawler.engine.downloader.middleware._rough_sizes == {}++ @pytest.mark.only_asyncio+ @coroutine_test+ async def test_same_request_downloaded_twice(self):+ """The rough size of a request object being downloaded twice at the same+ time is counted once, so that it is gone once both downloads finish."""++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {"RESPONSE_ROUGH_SIZE": 1024}++ async def parse(self, response):+ assert self.crawler.engine+ request = Request("data:,a")+ # Only one of the two downloads succeeds, the other one fails+ # because the request object is already being downloaded.+ await asyncio.gather(+ self.crawler.engine.download_async(request),+ self.crawler.engine.download_async(request),+ return_exceptions=True,+ )++ crawler = get_crawler(TestSpider)+ await maybe_deferred_to_future(crawler.crawl())++ assert crawler.engine+ middleware = crawler.engine.downloader.middleware+ assert middleware._rough_sizes == {}+ assert middleware._rough_active_size == 0++ @coroutine_test+ async def test_rough_size_triggers_backout(self):+ """Rough sizes of requests being downloaded count toward the limit, even+ if their responses turn out to be empty."""++ class SlowDown:+ """Keeps requests in flight long enough for the engine to check for+ backout while their rough size is being counted."""++ async def process_request(self, request):+ await sleep(0.01)++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,", "data:,"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {SlowDown: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 512,+ "RESPONSE_ROUGH_SIZE": 1024,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats+++class TestRequestBackout:+ @pytest.fixture(autouse=True)+ def use_caplog(self, caplog):+ self.caplog = caplog++ @coroutine_test+ async def test_none(self):++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 0++ assert _backout_stats(crawler) == {}++ @coroutine_test+ async def test_concurrency(self):++ class SlowDown:+ """Downloader middleware that returns a non-instant deferred from+ process_request, to force need_backout calls to happen at that+ point.++ The delay is deliberately non-zero so that the concurrency backout+ state lasts a measurable amount of wall-clock time, which keeps the+ request_backout_seconds/concurrency stat reliably above 0."""++ def process_request(self, request):+ from twisted.internet import reactor++ d: Deferred[None] = Deferred()+ reactor.callLater(0.01, d.callback, None)+ return d++ class TestSpider(Spider):+ name = "test"+ # Several start URLs so that, with CONCURRENT_REQUESTS=1, the engine+ # reliably attempts to schedule a second request while the first one+ # is still active, which is what triggers the concurrency backout.+ start_urls = ["data:,"] * 5+ custom_settings = {+ "CONCURRENT_REQUESTS": 1,+ "DOWNLOADER_MIDDLEWARES": {SlowDown: 0},+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 0++ expected_stats = {+ "request_backout_seconds/concurrency": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats++ @coroutine_test+ async def test_response_size(self):++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,a"]+ custom_settings = {+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats++ @pytest.mark.parametrize(+ ("status", "callback"),+ [+ (200, lambda response: response.css("a")),+ (200, _raise),+ (404, None),+ ],+ ids=["selector", "callback-exception", "errback"],+ )+ @coroutine_test+ async def test_response_max_active_size_no_cycles(self, status, callback):+ """Scrapy frees responses through reference counting alone, so a crawl+ does not stall on RESPONSE_MAX_ACTIVE_SIZE without garbage+ collection."""++ class ResponseMiddleware:+ def process_request(self, request):+ return HtmlResponse(request.url, status=status, body=b"<a></a>")++ class TestSpider(Spider):+ name = "test"+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {ResponseMiddleware: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ async def start(self):+ for n in range(3):+ yield Request(+ f"https://example.com/{n}",+ callback=callback,+ errback=lambda failure: None,+ )++ crawler = get_crawler(TestSpider)+ # The log capture of pytest keeps the tracebacks of logged errors, and+ # with them the responses they refer to.+ logging.disable()+ gc.disable()+ try:+ crawl_deferred = crawler.crawl()+ # A stall would otherwise hang the test.+ timeout = call_later(10, crawl_deferred.cancel)+ try:+ await maybe_deferred_to_future(crawl_deferred)+ finally:+ timeout.cancel()+ finally:+ gc.enable()+ logging.disable(logging.NOTSET)++ assert crawler.stats+ assert crawler.stats.get_value("response_received_count") == 3++ @coroutine_test+ async def test_response_size_process_request(self):++ class DownloaderMiddleware:+ def process_request(self, request):+ return Response("https://example.com", body=b"a")++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats++ @coroutine_test+ async def test_request_from_process_request(self):+ """A request returned from process_request does not count toward the+ limit, even though requests also have a body."""++ class DownloaderMiddleware:+ def __init__(self):+ self.replaced = False++ def process_request(self, request):+ if self.replaced:+ return None+ self.replaced = True+ return Request("data:,b", body=b"a" * 2000)++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 1000,+ "RESPONSE_ROUGH_SIZE": 0,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 0++ assert _backout_stats(crawler) == {}++ @coroutine_test+ async def test_response_size_process_response(self):++ class DownloaderMiddleware:+ def process_response(self, request, response):+ return Response("https://example.com", body=b"a")++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats++ @coroutine_test+ async def test_response_size_process_exception(self):++ class DownloaderMiddleware1:+ def process_exception(self, request, exception):+ return Response("https://example.com", body=b"a")++ class DownloaderMiddleware2:+ def process_request(self, request):+ raise ValueError++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {+ DownloaderMiddleware1: 0,+ DownloaderMiddleware2: 1,+ },+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ def parse(self, response):+ pass++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_stats++ @coroutine_test+ async def test_response_size_download(self):+ """Ensure that responses from engine.download calls are also taken into+ account for the RESPONSE_MAX_ACTIVE_SIZE setting."""++ class SlowDown:+ """Item pipeline that returns a non-instant deferred, to force+ need_backout calls to happen at that point."""++ def process_item(self, item):+ from twisted.internet import reactor++ d: Deferred[dict[Any, Any]] = Deferred()+ reactor.callLater(0, d.callback, {})+ return d++ class TestSpider(Spider):+ name = "test"+ start_urls = ["data:,"]+ custom_settings = {+ "ITEM_PIPELINES": {SlowDown: 0},+ "RESPONSE_MAX_ACTIVE_SIZE": 1,+ }++ async def parse(self, response):+ assert self.crawler.engine+ response = await self.crawler.engine.download_async(Request("data:,a"))+ yield {"response": response}++ crawler = get_crawler(TestSpider)+ self.caplog.clear()+ with self.caplog.at_level("INFO"):+ await maybe_deferred_to_future(crawler.crawl())++ assert _count_backout_logs(self.caplog) == 1++ expected_stats = {+ "request_backout_seconds/response_max_active_size": gt(0),+ "request_backout_seconds/total": gt(0),+ }+ assert _backout_stats(crawler) == expected_statsdiff --git a/tox.ini b/tox.iniindex 0feb4b3d5..6a0dc4888 100644--- a/tox.ini+++ b/tox.ini@@ -140,7 +140,7 @@ deps =# pytest 8.4.1 adds support for Twisted 25.5.0 but drops support for Twisted < 24.10.0pytest==8.4.0Protego==0.2.0- Twisted==21.7.0+ Twisted==23.8.0aiodns==3.3.0aioftp==0.26.1aiohttp==3.13.3@@ -207,7 +207,7 @@ basepython = {[min]basepython}deps ={[min]deps}Pillow==9.2.0- Twisted[http2]==21.7.0+ Twisted[http2]==23.8.0python-socks[asyncio]==2.6.0siosocks==0.2.0 # SOCKS server for testsboto3==1.35.2