Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Pause request processing based on the sum of body sizes of all active responses (#6430)

Adrián Chavesffe67f836f2b
diff --git a/docs/news/2.x.rst b/docs/news/2.x.rst
index dc0663207..6164b0cbc 100644
--- a/docs/news/2.x.rst
+++ b/docs/news/2.x.rst
@@ -4045,9 +4045,9 @@ New features
* A new ``DNS_RESOLVER`` setting allows enabling IPv6 support
(:gh:`1031`, :gh:`4227`)
-* A new :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` setting allows configuring
- the existing soft limit that pauses request downloads when the total
- response data being processed is too high (:gh:`1410`, :gh:`3551`)
+* A new ``SCRAPER_SLOT_MAX_ACTIVE_SIZE`` setting allows configuring the
+ existing soft limit that pauses request downloads when the total response
+ data being processed is too high (:gh:`1410`, :gh:`3551`)
* A new :setting:`TWISTED_REACTOR` setting allows customizing the
:mod:`~twisted.internet.reactor` that Scrapy uses, allowing to
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 84befb2be..56c4f2819 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -208,7 +208,8 @@ Engine API
==========
.. autoclass:: scrapy.core.engine.ExecutionEngine()
- :members: close_spider_async, needs_backout, running, scheduler, state
+ :members: close_spider_async, needs_backout, response_active_size, running,
+ scheduler, state
.. autoclass:: scrapy.core.engine.EngineState
:members:
diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst
index c1e88a32c..f1c0a019b 100644
--- a/docs/topics/optimize.rst
+++ b/docs/topics/optimize.rst
@@ -41,8 +41,7 @@ what every part of the engine is doing at a given moment:
len(engine.downloader.active) : 16
len(engine.scheduler.mqs) : 92
len(engine.scraper.slot.active) : 0
- engine.scraper.slot.active_size : 0
- engine.scraper.slot.needs_backout() : False
+ engine.response_active_size : 1310720
A coding agent using the :ref:`Scrapy MCP server <using-mcp-server>` can also
retrieve these values directly.
@@ -64,10 +63,13 @@ Take a few readings at different points of the crawl:
page at a time cannot use more concurrency than it creates. See
:ref:`optimize-requests`.
-- ``needs_backout()`` is ``True``, or ``active_size`` approaches
- :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`: responses arrive faster than your
- callbacks and :ref:`item pipelines <topics-item-pipeline>` handle them. The
- bottleneck is your own code.
+- :attr:`engine.response_active_size
+ <scrapy.core.engine.ExecutionEngine.response_active_size>` approaches
+ :setting:`RESPONSE_MAX_ACTIVE_SIZE`, or the
+ :stat:`request_backout_seconds/response_max_active_size` stat grows (see
+ :ref:`crawl-bottlenecks`): responses arrive faster than your callbacks and
+ :ref:`item pipelines <topics-item-pipeline>` handle them, or your code
+ keeps responses in memory. The bottleneck is your own code.
- ``len(engine.scheduler.mqs)`` grows without settling: the crawl discovers
requests faster than it downloads them. This is what makes long crawls run
@@ -252,7 +254,7 @@ Lowering resource usage
Lowering memory usage
---------------------
-- Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.
+- Lower :setting:`RESPONSE_MAX_ACTIVE_SIZE`.
- Lower :setting:`DOWNLOAD_MAXSIZE`, which allows a single response to take up
to 1 GiB of memory by default, multiplied by your concurrency. Set
@@ -373,7 +375,7 @@ For broad crawls, consider these adjustments:
(e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory
allow.
- - Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing
+ - Increase :setting:`RESPONSE_MAX_ACTIVE_SIZE` when increasing
:setting:`CONCURRENT_REQUESTS` stops making a difference.
- .. _broad-crawls-bfo:
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 0503cefab..bbfba806f 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -534,6 +534,31 @@ run:
Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple
modules by separating them with commas.
+.. _crawl-bottlenecks:
+
+Identifying crawl bottlenecks
+=============================
+
+The :stat:`request_backout_seconds/concurrency`,
+:stat:`request_backout_seconds/response_max_active_size` and
+:stat:`request_backout_seconds/total` stats show how long request scheduling
+was paused during a crawl, and why.
+
+For example, after a crawl you might see::
+
+ 2025-01-01 00:00:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
+ {'request_backout_seconds/concurrency': 12.5,
+ 'request_backout_seconds/response_max_active_size': 45.2,
+ 'request_backout_seconds/total': 57.7,
+ ...}
+
+In this case, the spider spent about 45 seconds paused due to large responses
+in memory. You could:
+
+- Increase :setting:`RESPONSE_MAX_ACTIVE_SIZE` if your machine has enough RAM.
+- Check that your code doesn't hold strong references to
+ :class:`~scrapy.http.Response` objects longer than necessary.
+
.. _bans:
Avoiding getting banned
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index c0d25ad94..a5316ca38 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -2079,6 +2079,53 @@ Adjust redirect request priority relative to original request:
- **a positive priority adjust (default) means higher priority.**
- a negative priority adjust means lower priority.
+.. setting:: RESPONSE_MAX_ACTIVE_SIZE
+
+RESPONSE_MAX_ACTIVE_SIZE
+------------------------
+
+.. versionadded:: VERSION
+
+Default: ``5_000_000``
+
+Soft limit (in bytes) on the total size of responses being kept in memory.
+
+This counts both the size of response bodies that have passed through
+:ref:`downloader middlewares <topics-downloader-middleware>` and remain in
+memory, and the :setting:`rough size <RESPONSE_ROUGH_SIZE>` of requests
+currently being downloaded.
+
+While the total is above this value, Scrapy pauses sending new requests to the
+downloader. A higher value improves crawl speed at the cost of memory usage.
+``0`` disables the limit.
+
+Scrapy logs an info-level message the first time the limit is reached. To see
+how long request processing was paused because of it over a whole crawl, check
+the :stat:`request_backout_seconds/response_max_active_size` stat.
+
+.. caution::
+
+ Responses that your code keeps a strong reference to, e.g. in the
+ :attr:`.Request.meta` of a scheduled request or in a component attribute,
+ count toward this limit until that reference is gone, so accumulating them
+ can pause a crawl indefinitely. So do responses in a reference cycle,
+ until Python's garbage collector frees them. To find out whether your
+ crawl keeps responses alive, see :ref:`topics-leaks`.
+
+.. setting:: RESPONSE_ROUGH_SIZE
+
+RESPONSE_ROUGH_SIZE
+-------------------
+
+.. versionadded:: VERSION
+
+Default: ``None``
+
+Estimated size (in bytes) to count toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`
+for each request being downloaded, whose actual response size is not known yet.
+
+``None`` lets Scrapy choose a value. ``0`` counts only responses.
+
.. setting:: ROBOTSTXT_OBEY
ROBOTSTXT_OBEY
@@ -2242,19 +2289,6 @@ For available choices, see :setting:`SCHEDULER_MEMORY_QUEUE`.
:start-after: queue-common-starts
:end-before: queue-common-ends
-
-.. setting:: SCRAPER_SLOT_MAX_ACTIVE_SIZE
-
-SCRAPER_SLOT_MAX_ACTIVE_SIZE
-----------------------------
-
-Default: ``5_000_000``
-
-Soft limit (in bytes) for response data being processed.
-
-While the sum of the sizes of all responses being processed is above this value,
-Scrapy does not process new requests.
-
.. setting:: SPIDER_CONTRACTS
SPIDER_CONTRACTS
diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst
index 8eb628c2d..cc1b12d9f 100644
--- a/docs/topics/signals.rst
+++ b/docs/topics/signals.rst
@@ -173,8 +173,8 @@ scheduler_empty
.. warning:: Only wait for this signal from
:meth:`~scrapy.Spider.start`. While no request can be sent, e.g. while
- the responses being parsed exceed
- :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`, the engine does not ask the
+ the responses in memory exceed
+ :setting:`RESPONSE_MAX_ACTIVE_SIZE`, the engine does not ask the
scheduler for requests, and hence this signal is not sent. So waiting
for it from a :ref:`callback <callbacks>` can hang the crawl,
because the response being parsed is itself one of the responses that
diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst
index 57b506a5e..1dd64297b 100644
--- a/docs/topics/stats.rst
+++ b/docs/topics/stats.rst
@@ -610,6 +610,24 @@ one per actual value of the placeholder.
Set by :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`.
+.. stat:: request_backout_seconds/concurrency
+
+``request_backout_seconds/concurrency``
+ Seconds during which sending requests to the downloader was paused because
+ :setting:`CONCURRENT_REQUESTS` was reached.
+
+.. stat:: request_backout_seconds/response_max_active_size
+
+``request_backout_seconds/response_max_active_size``
+ Seconds during which sending requests to the downloader was paused because
+ :setting:`RESPONSE_MAX_ACTIVE_SIZE` was reached.
+
+.. stat:: request_backout_seconds/total
+
+``request_backout_seconds/total``
+ Seconds during which sending requests to the downloader was paused, for
+ any of the reasons above.
+
.. stat:: request_depth_count/{depth}
``request_depth_count/{depth}``
diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst
index f23bdf37d..b4770219e 100644
--- a/docs/topics/telnetconsole.rst
+++ b/docs/topics/telnetconsole.rst
@@ -125,9 +125,8 @@ engine status::
len(engine.scheduler.mqs) : 92
len(engine.scraper.slot.queue) : 0
len(engine.scraper.slot.active) : 0
- engine.scraper.slot.active_size : 0
+ engine.response_active_size : 1310720
engine.scraper.slot.itemproc_size : 0
- engine.scraper.slot.needs_backout() : False
Pause, resume and stop the Scrapy engine
diff --git a/pyproject.toml b/pyproject.toml
index 064ce6a05..89355c49c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,7 +8,7 @@ dynamic = ["version"]
description = "A high-level Web Crawling and Web Scraping framework"
dependencies = [
"PyDispatcher>=2.0.5",
- "Twisted>=21.7.0",
+ "Twisted>=23.8.0",
"aiodns>=3.3.0",
"aioftp>=0.26.1",
"aiohttp>=3.13.3",
@@ -79,7 +79,7 @@ ptpython = ["ptpython>=3.0.23"]
robotparser = ["robotexclusionrulesparser>=1.6.2"]
s3 = ["aioboto3>=9.0.0", "boto3>=1.35.2"]
socks = ["python-socks[asyncio]>=2.6.0"]
-twisted-http2 = ["Twisted[http2]>=21.7.0"]
+twisted-http2 = ["Twisted[http2]>=23.8.0"]
uvloop = [
"uvloop>=0.17.0; platform_system != 'Windows'",
]
diff --git a/scrapy/core/downloader/__init__.py b/scrapy/core/downloader/__init__.py
index f851ed745..babb9a010 100644
--- a/scrapy/core/downloader/__init__.py
+++ b/scrapy/core/downloader/__init__.py
@@ -5,7 +5,8 @@ import random
import warnings
from collections import deque
from dataclasses import dataclass, field
-from time import monotonic, time
+from logging import getLogger
+from time import monotonic, perf_counter, time
from typing import TYPE_CHECKING, Any
from twisted.internet.defer import Deferred, DeferredList, inlineCallbacks
@@ -42,6 +43,8 @@ if TYPE_CHECKING:
from scrapy.settings import BaseSettings
from scrapy.signalmanager import SignalManager
+logger = getLogger(__name__)
+
# Request.meta key holding the delay of a request and the time before which it
# must not be sent because of it. The deadline is a wall-clock time, so that it
@@ -233,6 +236,11 @@ class Downloader:
self.per_slot_settings: dict[str, dict[str, Any]] = self.settings.getdict(
"DOWNLOAD_SLOTS"
)
+ self._stats = crawler.stats
+ # (reason, start_time): current backout reason and when it began, or
+ # (None, None) if not backing out.
+ self._last_backout: tuple[str | None, float | None] = (None, None)
+ self._max_active_size_warned = False
@property
def randomize_delay(self) -> bool:
@@ -250,6 +258,7 @@ class Downloader:
self, request: Request, spider: Spider | None = None
) -> Generator[Deferred[Any], Any, Response | Request]:
self.active.add(request)
+ self.middleware._count_rough_size(request)
try:
result: Response | Request = yield (
deferred_from_coro(
@@ -259,10 +268,50 @@ class Downloader:
return result
finally:
self.active.remove(request)
+ self.middleware._discount_rough_size(request)
+
+ def _record_backout(self, reason: str | None) -> None:
+ last_reason, last_reason_start_time = self._last_backout
+ if last_reason == reason:
+ return
+ current_time = perf_counter()
+ if last_reason is not None and self._stats is not None:
+ assert last_reason_start_time is not None
+ last_reason_seconds = current_time - last_reason_start_time
+ self._stats.inc_value("request_backout_seconds/total", last_reason_seconds)
+ self._stats.inc_value(
+ f"request_backout_seconds/{last_reason}", last_reason_seconds
+ )
+ self._last_backout = (reason, current_time)
def needs_backout(self) -> bool:
# A total concurrency of 0 means no limit.
- return 0 < self.total_concurrency <= len(self.active)
+ if 0 < self.total_concurrency <= len(self.active):
+ self._record_backout("concurrency")
+ return True
+ if self._exceeds_max_active_size():
+ self._record_backout("response_max_active_size")
+ if not self._max_active_size_warned:
+ self._max_active_size_warned = True
+ logger.info(
+ f"Pausing request processing: the active response size "
+ f"({self.middleware._total_active_size} B) has reached "
+ f"RESPONSE_MAX_ACTIVE_SIZE "
+ f"({self.middleware._max_active_size} B). See "
+ f"https://docs.scrapy.org/en/latest/topics/settings.html#response-max-active-size "
+ f"and the request_backout_seconds/response_max_active_size "
+ f"stat. This message is only logged once."
+ )
+ return True
+ self._record_backout(None)
+ return False
+
+ def _exceeds_max_active_size(self) -> bool:
+ max_active_size = self.middleware._max_active_size
+ return (
+ bool(max_active_size)
+ and self.middleware._total_active_size >= max_active_size
+ )
@_warn_spider_arg
def _get_slot(
@@ -489,6 +538,7 @@ class Downloader:
self._stop_slot_gc()
for slot in self.slots.values():
slot.close()
+ self._record_backout(None)
async def _close_async(self) -> None:
"""Close the downloader and its download handlers."""
diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py
index 14e11516e..8c316f6fd 100644
--- a/scrapy/core/downloader/middleware.py
+++ b/scrapy/core/downloader/middleware.py
@@ -7,12 +7,13 @@ See documentation in docs/topics/downloader-middleware.rst
from __future__ import annotations
import warnings
-from functools import wraps
+from functools import partial, wraps
from itertools import islice
from typing import TYPE_CHECKING, Any
+from weakref import WeakSet, finalize
from scrapy.exceptions import CloseSpider, ScrapyDeprecationWarning, _InvalidOutput
-from scrapy.http import Request, Response
+from scrapy.http import BodyFileResponse, Request, Response
from scrapy.middleware import MiddlewareManager
from scrapy.utils.conf import build_component_list
from scrapy.utils.defer import (
@@ -33,15 +34,62 @@ if TYPE_CHECKING:
from scrapy.settings import BaseSettings
+def _get_max_active_size(settings: BaseSettings) -> int:
+ deprecated_priority = settings.getpriority("SCRAPER_SLOT_MAX_ACTIVE_SIZE")
+ priority = settings.getpriority("RESPONSE_MAX_ACTIVE_SIZE")
+ assert deprecated_priority is not None
+ assert priority is not None
+ if deprecated_priority <= 0:
+ return settings.getint("RESPONSE_MAX_ACTIVE_SIZE")
+ if priority >= deprecated_priority:
+ warnings.warn(
+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated and is "
+ "being ignored because RESPONSE_MAX_ACTIVE_SIZE is set with an "
+ "equal or higher priority. Remove SCRAPER_SLOT_MAX_ACTIVE_SIZE "
+ "from your settings.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return settings.getint("RESPONSE_MAX_ACTIVE_SIZE")
+ warnings.warn(
+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated, use "
+ "RESPONSE_MAX_ACTIVE_SIZE instead.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE")
+
+
+def _get_response_rough_size(settings: BaseSettings, max_active_size: int) -> int:
+ if settings.get("RESPONSE_ROUGH_SIZE") is not None:
+ return settings.getint("RESPONSE_ROUGH_SIZE")
+ concurrency = settings.getint("CONCURRENT_REQUESTS")
+ if not concurrency:
+ # Unlimited concurrency: there is no bound on the number of in-flight
+ # requests to spread a share of the limit over.
+ return 0
+ # Requests being downloaded, whose response size is unknown, may take up to
+ # a quarter of the limit; the rest is for responses already in memory.
+ return max_active_size // (4 * concurrency)
+
+
class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstract-method
component_name = "downloader middleware"
def __init__(self, *middlewares: Any, crawler: Crawler | None = None) -> None:
super().__init__(*middlewares, crawler=crawler)
- self._response_exceptions = (
- crawler.settings.getbool("DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS")
- if crawler is not None
- else False
+ assert crawler is not None
+ settings = crawler.settings
+ self._max_active_size: int = _get_max_active_size(settings)
+ self._response_rough_size: int = _get_response_rough_size(
+ settings, self._max_active_size
+ )
+ self._response_active_size = 0
+ self._tracked_responses: WeakSet[Response] = WeakSet()
+ self._rough_active_size = 0
+ self._rough_sizes: dict[Request, int] = {}
+ self._response_exceptions = settings.getbool(
+ "DOWNLOADER_MIDDLEWARE_RESPONSE_EXCEPTIONS"
)
@classmethod
@@ -63,6 +111,39 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstrac
if process_exception is not None:
self._check_mw_method_spider_arg(process_exception)
+ @property
+ def _total_active_size(self) -> int:
+ return self._response_active_size + self._rough_active_size
+
+ def _count_rough_size(self, request: Request) -> None:
+ # Counting the same request twice, which concurrent downloads of the
+ # same request object would do, would leak its rough size, since only
+ # one discount call can find it.
+ if request in self._rough_sizes:
+ return
+ self._rough_sizes[request] = self._response_rough_size
+ self._rough_active_size += self._response_rough_size
+
+ def _discount_rough_size(self, request: Request) -> None:
+ self._rough_active_size -= self._rough_sizes.pop(request, 0)
+
+ def _count_response_size(self, response: Response, request: Request) -> None:
+ # The actual response size replaces the rough size estimate of its
+ # request.
+ self._discount_rough_size(request)
+ if (
+ isinstance(response, BodyFileResponse)
+ or response in self._tracked_responses
+ ):
+ return
+ self._tracked_responses.add(response)
+ size = len(response.body)
+ self._response_active_size += size
+ finalize(response, partial(self._discount_response_size, size))
+
+ def _discount_response_size(self, size: int) -> None:
+ self._response_active_size -= size
+
def download(
self,
download_func: Callable[[Request, Spider], Deferred[Response]],
@@ -122,8 +203,13 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstrac
f"Request, got {response.__class__.__name__}"
)
if response:
+ if isinstance(response, Response):
+ self._count_response_size(response, request)
return response
- return await download_func(request)
+ response = await download_func(request)
+ if response is not None:
+ self._count_response_size(response, request)
+ return response
async def _process_response(
self, response: Response | Request, request: Request, index: int = 0
@@ -160,6 +246,8 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstrac
f"got {type(result)}"
)
response = result
+ if isinstance(response, Response):
+ self._count_response_size(response, request)
return response
def _warn_response_exception(
@@ -226,5 +314,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): # pylint: disable=abstrac
f"Request, got {type(response)}"
)
if response:
+ if isinstance(response, Response):
+ self._count_response_size(response, request)
return response, index
raise exception
diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py
index 790718ddd..ded2ba30e 100644
--- a/scrapy/core/engine.py
+++ b/scrapy/core/engine.py
@@ -278,6 +278,14 @@ class ExecutionEngine:
"""
return self._state
+ @property
+ def response_active_size(self) -> int:
+ """Size, in bytes, that counts toward :setting:`RESPONSE_MAX_ACTIVE_SIZE`.
+
+ .. versionadded:: VERSION
+ """
+ return self.downloader.middleware._total_active_size
+
@property
def running(self) -> bool:
"""Whether the engine is in the :attr:`EngineState.RUNNING` state.
@@ -511,7 +519,6 @@ class ExecutionEngine:
or not self._slot
or bool(self._slot.closing)
or self.downloader.needs_backout()
- or self.scraper.slot.needs_backout()
)
def _remove_request(self, _: Any, request: Request) -> None:
diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py
index c1915436d..bdf3d5801 100644
--- a/scrapy/core/scraper.py
+++ b/scrapy/core/scraper.py
@@ -20,7 +20,7 @@ from scrapy.exceptions import (
IgnoreRequest,
ScrapyDeprecationWarning,
)
-from scrapy.http import BodyFileResponse, Request, Response
+from scrapy.http import Request, Response
from scrapy.http.request import _active_crawler_scope
from scrapy.pipelines import ItemPipelineManager
from scrapy.utils.asyncgen import as_async_generator
@@ -76,16 +76,13 @@ async def _aset_parent_id(
yield o
-class Slot:
+class _Slot:
"""Scraper slot (one per running spider)"""
- MIN_RESPONSE_SIZE = 1024
-
- def __init__(self, max_active_size: int = 5000000):
- self.max_active_size: int = max_active_size
+ def __init__(self, crawler: Crawler) -> None:
+ self._crawler = crawler
self.queue: deque[QueueTuple] = deque()
self.active: set[Request] = set()
- self.active_size: int = 0
self.itemproc_size: int = 0
self.closing: Deferred[Spider] | None = None
@@ -95,10 +92,6 @@ class Slot:
# this Deferred will be awaited in enqueue_scrape()
deferred: Deferred[None] = Deferred()
self.queue.append((result, request, deferred))
- if isinstance(result, Response) and not isinstance(result, BodyFileResponse):
- self.active_size += max(len(result.body), self.MIN_RESPONSE_SIZE)
- else:
- self.active_size += self.MIN_RESPONSE_SIZE
return deferred
def next_response_request_deferred(self) -> QueueTuple:
@@ -106,23 +99,48 @@ class Slot:
self.active.add(request)
return result, request, deferred
- def finish_response(self, result: Response | Failure, request: Request) -> None:
+ def finish_response(self, request: Request) -> None:
self.active.remove(request)
- if isinstance(result, Response) and not isinstance(result, BodyFileResponse):
- self.active_size -= max(len(result.body), self.MIN_RESPONSE_SIZE)
- else:
- self.active_size -= self.MIN_RESPONSE_SIZE
def is_idle(self) -> bool:
return not (self.queue or self.active or self.itemproc_size)
+ @property
+ def active_size(self) -> int:
+ warnings.warn(
+ "Scraper.slot.active_size is deprecated, use "
+ "ExecutionEngine.response_active_size instead.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ assert self._crawler.engine is not None
+ return self._crawler.engine.response_active_size
+
+ @property
+ def max_active_size(self) -> int:
+ warnings.warn(
+ "Scraper.slot.max_active_size is deprecated, use the "
+ "RESPONSE_MAX_ACTIVE_SIZE setting instead.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ assert self._crawler.engine is not None
+ return self._crawler.engine.downloader.middleware._max_active_size
+
def needs_backout(self) -> bool:
- return self.active_size > self.max_active_size
+ warnings.warn(
+ "Scraper.slot.needs_backout() is deprecated, use "
+ "ExecutionEngine.needs_backout() instead.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ assert self._crawler.engine is not None
+ return self._crawler.engine.downloader._exceeds_max_active_size()
class Scraper:
def __init__(self, crawler: Crawler) -> None:
- self.slot: Slot | None = None
+ self.slot: _Slot | None = None
self.spidermw: SpiderMiddlewareManager = build_from_crawler(
SpiderMiddlewareManager, crawler
)
@@ -185,7 +203,7 @@ class Scraper:
.. versionadded:: 2.14
"""
- self.slot = Slot(self.crawler.settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE"))
+ self.slot = _Slot(self.crawler)
if not self.crawler.spider:
raise RuntimeError(
"Scraper.open_spider() called before Crawler.spider is set."
@@ -250,7 +268,7 @@ class Scraper:
extra={"spider": self.crawler.spider},
)
finally:
- self.slot.finish_response(result, request)
+ self.slot.finish_response(request)
self._check_if_closing()
self._scrape_next()
@@ -602,3 +620,14 @@ class Scraper:
finally:
self.slot.itemproc_size -= 1
self._check_if_closing()
+
+
+def __getattr__(name: str) -> Any:
+ if name == "Slot":
+ warnings.warn(
+ "scrapy.core.scraper.Slot is deprecated.",
+ ScrapyDeprecationWarning,
+ stacklevel=2,
+ )
+ return _Slot
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 69560bc67..0c162114b 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -201,6 +201,8 @@ __all__ = [
"REMOTE_CONTROL_TIMEOUT_MAX",
"REMOTE_CONTROL_TRACEBACK_MAX_BYTES",
"REQUEST_FINGERPRINTER_CLASS",
+ "RESPONSE_MAX_ACTIVE_SIZE",
+ "RESPONSE_ROUGH_SIZE",
"RETRY_ENABLED",
"RETRY_EXCEPTIONS",
"RETRY_GIVE_UP_LOG_LEVEL",
@@ -605,7 +607,9 @@ SCHEDULER_PRIORITY_QUEUE = "scrapy.pqueues.DownloaderAwarePriorityQueue"
SCHEDULER_START_DISK_QUEUE = "scrapy.squeues.PickleFifoDiskQueue"
SCHEDULER_START_MEMORY_QUEUE = "scrapy.squeues.FifoMemoryQueue"
-SCRAPER_SLOT_MAX_ACTIVE_SIZE = 5000000
+SCRAPER_SLOT_MAX_ACTIVE_SIZE = 5_000_000
+RESPONSE_MAX_ACTIVE_SIZE = 5_000_000
+RESPONSE_ROUGH_SIZE = None
SPIDER_CONTRACTS: dict[str, int] = {}
SPIDER_CONTRACTS_BASE = {
diff --git a/scrapy/statscollectors.py b/scrapy/statscollectors.py
index f012109db..fdb2986de 100644
--- a/scrapy/statscollectors.py
+++ b/scrapy/statscollectors.py
@@ -70,7 +70,11 @@ class StatsCollector(MutableMapping[str, Any]):
self._stats = stats
def inc_value(
- self, key: str, count: int = 1, start: int = 0, spider: Spider | None = None
+ self,
+ key: str,
+ count: float = 1,
+ start: float = 0,
+ spider: Spider | None = None,
) -> None:
"""Increment the *key* stat by *count*, or set it to *start* if it is
not set."""
@@ -176,7 +180,11 @@ class DummyStatsCollector(StatsCollector):
pass
def inc_value(
- self, key: str, count: int = 1, start: int = 0, spider: Spider | None = None
+ self,
+ key: str,
+ count: float = 1,
+ start: float = 0,
+ spider: Spider | None = None,
) -> None:
pass
diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py
index 2b1d68709..603667b4d 100644
--- a/scrapy/utils/defer.py
+++ b/scrapy/utils/defer.py
@@ -239,6 +239,10 @@ class _ParallelTasks:
# Only a running task can start another one, so once none is left no
# more work can come.
if not self._running:
+ # The work usually references self.start, so keeping it would
+ # leave a reference cycle that keeps the work arguments, e.g. a
+ # response, alive until the next garbage collection.
+ del self._work
self.finished.callback(None)
diff --git a/scrapy/utils/engine.py b/scrapy/utils/engine.py
index 2261bfe58..3ca3ec12e 100644
--- a/scrapy/utils/engine.py
+++ b/scrapy/utils/engine.py
@@ -24,9 +24,8 @@ def get_engine_status(engine: ExecutionEngine) -> list[tuple[str, object]]:
"len(engine.scheduler.mqs)",
"len(engine.scraper.slot.queue)",
"len(engine.scraper.slot.active)",
- "engine.scraper.slot.active_size",
+ "engine.response_active_size",
"engine.scraper.slot.itemproc_size",
- "engine.scraper.slot.needs_backout()",
]
checks: list[tuple[str, object]] = []
diff --git a/tests/test_core_scraper.py b/tests/test_core_scraper.py
index c4732ff3f..f5b5c7dc2 100644
--- a/tests/test_core_scraper.py
+++ b/tests/test_core_scraper.py
@@ -6,8 +6,9 @@ from typing import TYPE_CHECKING, Any, cast
import pytest
from scrapy import Request, Spider
-from scrapy.core.scraper import Scraper
-from scrapy.exceptions import IgnoreRequest
+from scrapy.core import scraper as scraper_module
+from scrapy.core.scraper import Scraper, _Slot
+from scrapy.exceptions import IgnoreRequest, ScrapyDeprecationWarning
from scrapy.http import Response
from scrapy.utils.defer import maybe_deferred_to_future
from scrapy.utils.spider import DefaultSpider
@@ -137,3 +138,45 @@ async def test_none_in_callback_output() -> None:
)
await crawler.crawl_async()
assert crawler.stats.get_value("item_scraped_count") == 2
+
+
+@coroutine_test
+async def test_slot_deprecated_attributes() -> None:
+ values: dict[str, Any] = {}
+
+ class SizeSpider(Spider):
+ name = "size"
+
+ async def start(self) -> AsyncIterator[Any]:
+ yield Request("data:,a")
+
+ def parse(self, response: Response) -> None:
+ engine = self.crawler.engine
+ assert engine is not None
+ slot = engine.scraper.slot
+ assert slot is not None
+ values["response_active_size"] = engine.response_active_size
+ for name in ("active_size", "max_active_size"):
+ with pytest.warns(ScrapyDeprecationWarning, match=name):
+ values[name] = getattr(slot, name)
+ with pytest.warns(ScrapyDeprecationWarning, match="needs_backout"):
+ values["needs_backout"] = slot.needs_backout()
+
+ crawler = get_crawler(SizeSpider, {"RESPONSE_MAX_ACTIVE_SIZE": 1})
+ await crawler.crawl_async()
+ assert values == {
+ "response_active_size": 1,
+ "active_size": 1,
+ "max_active_size": 1,
+ "needs_backout": True,
+ }
+
+
+def test_slot_deprecated() -> None:
+ with pytest.warns(ScrapyDeprecationWarning, match="Slot is deprecated"):
+ assert scraper_module.Slot is _Slot
+
+
+def test_missing_attribute() -> None:
+ with pytest.raises(AttributeError):
+ scraper_module.Foo
diff --git a/tests/test_downloader.py b/tests/test_downloader.py
new file mode 100644
index 000000000..ed123efef
--- /dev/null
+++ b/tests/test_downloader.py
@@ -0,0 +1,658 @@
+import asyncio
+import gc
+import logging
+import warnings
+from typing import Any
+
+import pytest
+from twisted.internet.defer import Deferred
+
+from scrapy import Request, Spider
+from scrapy.crawler import Crawler
+from scrapy.exceptions import ScrapyDeprecationWarning
+from scrapy.http import HtmlResponse, Response
+from scrapy.utils.asyncio import call_later, sleep
+from scrapy.utils.defer import maybe_deferred_to_future
+from scrapy.utils.test import get_crawler
+from tests.utils.decorators import coroutine_test
+
+
+def _count_backout_logs(caplog: pytest.LogCaptureFixture) -> int:
+ return sum(
+ 1
+ for record in caplog.records
+ if record.levelname == "INFO"
+ and str(record.msg).startswith("Pausing request processing")
+ )
+
+
+def _backout_stats(crawler: Crawler) -> dict[str, Any]:
+ assert crawler.stats
+ return {
+ k: v
+ for k, v in crawler.stats.get_stats().items()
+ if k.startswith("request_backout_seconds/")
+ }
+
+
+class OfflineSpider(Spider):
+ name = "offline"
+ start_urls = ["data:,"]
+
+ def parse(self, response):
+ pass
+
+
+def _assert_scraper_slot_deprecation(
+ warning_messages: pytest.WarningsRecorder, *, ignored: bool = False
+) -> None:
+ """Assert that a crawl emitted exactly one Scrapy deprecation warning, the
+ one about SCRAPER_SLOT_MAX_ACTIVE_SIZE.
+
+ Only Scrapy deprecation warnings are counted: a crawl may emit unrelated
+ warnings (e.g. a ResourceWarning for a socket garbage-collected while the
+ recorder is active), and those must not make the assertion flaky.
+
+ Pass ``ignored=True`` when RESPONSE_MAX_ACTIVE_SIZE is set with an equal or
+ higher priority and therefore SCRAPER_SLOT_MAX_ACTIVE_SIZE is being
+ ignored."""
+ deprecations = [
+ message
+ for message in warning_messages
+ if issubclass(message.category, ScrapyDeprecationWarning)
+ ]
+ assert len(deprecations) == 1
+ if ignored:
+ assert str(deprecations[0].message) == (
+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated and is "
+ "being ignored because RESPONSE_MAX_ACTIVE_SIZE is set with an "
+ "equal or higher priority. Remove SCRAPER_SLOT_MAX_ACTIVE_SIZE "
+ "from your settings."
+ )
+ else:
+ assert str(deprecations[0].message) == (
+ "The SCRAPER_SLOT_MAX_ACTIVE_SIZE setting is deprecated, use "
+ "RESPONSE_MAX_ACTIVE_SIZE instead."
+ )
+
+
+def _raise(response: Response) -> None:
+ raise ValueError(response.url)
+
+
+class gt:
+ __hash__ = None # type: ignore[assignment]
+
+ def __init__(self, value: float):
+ self.value = value
+
+ def __eq__(self, other: object) -> bool:
+ return isinstance(other, (int, float)) and other > self.value
+
+ def __repr__(self) -> str:
+ return f">{self.value}"
+
+
+class TestResponseMaxActiveSize:
+ @coroutine_test
+ async def test_default(self):
+ """A crawl without custom settings has its effective response max
+ active size set to 5 000 000, and triggers no deprecation warning."""
+ crawler = get_crawler(OfflineSpider)
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", ScrapyDeprecationWarning)
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 5_000_000
+
+ @coroutine_test
+ async def test_custom(self):
+ """Setting RESPONSE_MAX_ACTIVE_SIZE to a custom value changes the
+ effective response max active size."""
+ crawler = get_crawler(
+ OfflineSpider, settings_dict={"RESPONSE_MAX_ACTIVE_SIZE": 0}
+ )
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", ScrapyDeprecationWarning)
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 0
+
+ @coroutine_test
+ async def test_deprecated_default(self):
+ """Setting SCRAPER_SLOT_MAX_ACTIVE_SIZE triggers a deprecation warning,
+ even if it is the default value."""
+ crawler = get_crawler(
+ OfflineSpider, settings_dict={"SCRAPER_SLOT_MAX_ACTIVE_SIZE": 5_000_000}
+ )
+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 5_000_000
+ _assert_scraper_slot_deprecation(warning_messages)
+
+ @coroutine_test
+ async def test_deprecated_custom(self):
+ """Setting SCRAPER_SLOT_MAX_ACTIVE_SIZE to a custom value triggers a
+ deprecation warning, and changes the effective response max active
+ size."""
+ crawler = get_crawler(
+ OfflineSpider, settings_dict={"SCRAPER_SLOT_MAX_ACTIVE_SIZE": 0}
+ )
+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 0
+ _assert_scraper_slot_deprecation(warning_messages)
+
+ @coroutine_test
+ async def test_both(self):
+ """Setting RESPONSE_MAX_ACTIVE_SIZE and SCRAPER_SLOT_MAX_ACTIVE_SIZE to
+ different values with the same setting priority triggers a deprecation
+ warning about SCRAPER_SLOT_MAX_ACTIVE_SIZE being ignored, and makes
+ the value of RESPONSE_MAX_ACTIVE_SIZE the effective response max active
+ size."""
+ crawler = get_crawler(
+ OfflineSpider,
+ settings_dict={
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ "SCRAPER_SLOT_MAX_ACTIVE_SIZE": 2,
+ },
+ )
+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 1
+ _assert_scraper_slot_deprecation(warning_messages, ignored=True)
+
+ @coroutine_test
+ async def test_both_deprecated_priority(self):
+ """Setting RESPONSE_MAX_ACTIVE_SIZE and SCRAPER_SLOT_MAX_ACTIVE_SIZE to
+ different values and SCRAPER_SLOT_MAX_ACTIVE_SIZE with a higher
+ priority triggers a deprecation warning about
+ SCRAPER_SLOT_MAX_ACTIVE_SIZE but also makes the value of
+ SCRAPER_SLOT_MAX_ACTIVE_SIZE the effective response max active size."""
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+
+ @classmethod
+ def update_settings(cls, settings):
+ settings.set("RESPONSE_MAX_ACTIVE_SIZE", 1, priority=100)
+ settings.set("SCRAPER_SLOT_MAX_ACTIVE_SIZE", 2, priority=101)
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ with pytest.warns(ScrapyDeprecationWarning) as warning_messages:
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._max_active_size == 2
+ _assert_scraper_slot_deprecation(warning_messages)
+
+
+class TestResponseRoughSize:
+ @pytest.fixture(autouse=True)
+ def use_caplog(self, caplog):
+ self.caplog = caplog
+
+ @pytest.mark.parametrize(
+ ("settings_dict", "expected"),
+ [
+ # A quarter of RESPONSE_MAX_ACTIVE_SIZE split among
+ # CONCURRENT_REQUESTS requests.
+ ({}, 78125),
+ ({"CONCURRENT_REQUESTS": 100}, 12500),
+ ({"RESPONSE_MAX_ACTIVE_SIZE": 8_000_000}, 125_000),
+ # Unlimited concurrency leaves no number of requests to split it
+ # among.
+ ({"CONCURRENT_REQUESTS": 0}, 0),
+ ({"RESPONSE_MAX_ACTIVE_SIZE": 0}, 0),
+ ({"RESPONSE_ROUGH_SIZE": 1}, 1),
+ ({"RESPONSE_ROUGH_SIZE": 0}, 0),
+ ],
+ )
+ @coroutine_test
+ async def test_value(self, settings_dict, expected):
+ crawler = get_crawler(OfflineSpider, settings_dict=settings_dict)
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", ScrapyDeprecationWarning)
+ await maybe_deferred_to_future(crawler.crawl())
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._response_rough_size == expected
+
+ @coroutine_test
+ async def test_response_replaces_rough_size(self):
+ """The rough size of a request stops counting as soon as the size of its
+ response is known."""
+ sizes: list[tuple[int, int]] = []
+
+ class DownloaderMiddleware:
+ def __init__(self, crawler):
+ self.crawler = crawler
+
+ @classmethod
+ def from_crawler(cls, crawler):
+ return cls(crawler)
+
+ def process_response(self, request, response):
+ middleware = self.crawler.engine.downloader.middleware
+ sizes.append(
+ (middleware._rough_active_size, middleware._response_active_size)
+ )
+ return response
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,a"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert sizes == [(0, 1)]
+ assert crawler.engine
+ assert crawler.engine.downloader.middleware._rough_sizes == {}
+
+ @pytest.mark.only_asyncio
+ @coroutine_test
+ async def test_same_request_downloaded_twice(self):
+ """The rough size of a request object being downloaded twice at the same
+ time is counted once, so that it is gone once both downloads finish."""
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {"RESPONSE_ROUGH_SIZE": 1024}
+
+ async def parse(self, response):
+ assert self.crawler.engine
+ request = Request("data:,a")
+ # Only one of the two downloads succeeds, the other one fails
+ # because the request object is already being downloaded.
+ await asyncio.gather(
+ self.crawler.engine.download_async(request),
+ self.crawler.engine.download_async(request),
+ return_exceptions=True,
+ )
+
+ crawler = get_crawler(TestSpider)
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert crawler.engine
+ middleware = crawler.engine.downloader.middleware
+ assert middleware._rough_sizes == {}
+ assert middleware._rough_active_size == 0
+
+ @coroutine_test
+ async def test_rough_size_triggers_backout(self):
+ """Rough sizes of requests being downloaded count toward the limit, even
+ if their responses turn out to be empty."""
+
+ class SlowDown:
+ """Keeps requests in flight long enough for the engine to check for
+ backout while their rough size is being counted."""
+
+ async def process_request(self, request):
+ await sleep(0.01)
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,", "data:,"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {SlowDown: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 512,
+ "RESPONSE_ROUGH_SIZE": 1024,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+
+class TestRequestBackout:
+ @pytest.fixture(autouse=True)
+ def use_caplog(self, caplog):
+ self.caplog = caplog
+
+ @coroutine_test
+ async def test_none(self):
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 0
+
+ assert _backout_stats(crawler) == {}
+
+ @coroutine_test
+ async def test_concurrency(self):
+
+ class SlowDown:
+ """Downloader middleware that returns a non-instant deferred from
+ process_request, to force need_backout calls to happen at that
+ point.
+
+ The delay is deliberately non-zero so that the concurrency backout
+ state lasts a measurable amount of wall-clock time, which keeps the
+ request_backout_seconds/concurrency stat reliably above 0."""
+
+ def process_request(self, request):
+ from twisted.internet import reactor
+
+ d: Deferred[None] = Deferred()
+ reactor.callLater(0.01, d.callback, None)
+ return d
+
+ class TestSpider(Spider):
+ name = "test"
+ # Several start URLs so that, with CONCURRENT_REQUESTS=1, the engine
+ # reliably attempts to schedule a second request while the first one
+ # is still active, which is what triggers the concurrency backout.
+ start_urls = ["data:,"] * 5
+ custom_settings = {
+ "CONCURRENT_REQUESTS": 1,
+ "DOWNLOADER_MIDDLEWARES": {SlowDown: 0},
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 0
+
+ expected_stats = {
+ "request_backout_seconds/concurrency": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+ @coroutine_test
+ async def test_response_size(self):
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,a"]
+ custom_settings = {
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+ @pytest.mark.parametrize(
+ ("status", "callback"),
+ [
+ (200, lambda response: response.css("a")),
+ (200, _raise),
+ (404, None),
+ ],
+ ids=["selector", "callback-exception", "errback"],
+ )
+ @coroutine_test
+ async def test_response_max_active_size_no_cycles(self, status, callback):
+ """Scrapy frees responses through reference counting alone, so a crawl
+ does not stall on RESPONSE_MAX_ACTIVE_SIZE without garbage
+ collection."""
+
+ class ResponseMiddleware:
+ def process_request(self, request):
+ return HtmlResponse(request.url, status=status, body=b"<a></a>")
+
+ class TestSpider(Spider):
+ name = "test"
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {ResponseMiddleware: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ async def start(self):
+ for n in range(3):
+ yield Request(
+ f"https://example.com/{n}",
+ callback=callback,
+ errback=lambda failure: None,
+ )
+
+ crawler = get_crawler(TestSpider)
+ # The log capture of pytest keeps the tracebacks of logged errors, and
+ # with them the responses they refer to.
+ logging.disable()
+ gc.disable()
+ try:
+ crawl_deferred = crawler.crawl()
+ # A stall would otherwise hang the test.
+ timeout = call_later(10, crawl_deferred.cancel)
+ try:
+ await maybe_deferred_to_future(crawl_deferred)
+ finally:
+ timeout.cancel()
+ finally:
+ gc.enable()
+ logging.disable(logging.NOTSET)
+
+ assert crawler.stats
+ assert crawler.stats.get_value("response_received_count") == 3
+
+ @coroutine_test
+ async def test_response_size_process_request(self):
+
+ class DownloaderMiddleware:
+ def process_request(self, request):
+ return Response("https://example.com", body=b"a")
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+ @coroutine_test
+ async def test_request_from_process_request(self):
+ """A request returned from process_request does not count toward the
+ limit, even though requests also have a body."""
+
+ class DownloaderMiddleware:
+ def __init__(self):
+ self.replaced = False
+
+ def process_request(self, request):
+ if self.replaced:
+ return None
+ self.replaced = True
+ return Request("data:,b", body=b"a" * 2000)
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 1000,
+ "RESPONSE_ROUGH_SIZE": 0,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 0
+
+ assert _backout_stats(crawler) == {}
+
+ @coroutine_test
+ async def test_response_size_process_response(self):
+
+ class DownloaderMiddleware:
+ def process_response(self, request, response):
+ return Response("https://example.com", body=b"a")
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {DownloaderMiddleware: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+ @coroutine_test
+ async def test_response_size_process_exception(self):
+
+ class DownloaderMiddleware1:
+ def process_exception(self, request, exception):
+ return Response("https://example.com", body=b"a")
+
+ class DownloaderMiddleware2:
+ def process_request(self, request):
+ raise ValueError
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {
+ "DOWNLOADER_MIDDLEWARES": {
+ DownloaderMiddleware1: 0,
+ DownloaderMiddleware2: 1,
+ },
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ def parse(self, response):
+ pass
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
+
+ @coroutine_test
+ async def test_response_size_download(self):
+ """Ensure that responses from engine.download calls are also taken into
+ account for the RESPONSE_MAX_ACTIVE_SIZE setting."""
+
+ class SlowDown:
+ """Item pipeline that returns a non-instant deferred, to force
+ need_backout calls to happen at that point."""
+
+ def process_item(self, item):
+ from twisted.internet import reactor
+
+ d: Deferred[dict[Any, Any]] = Deferred()
+ reactor.callLater(0, d.callback, {})
+ return d
+
+ class TestSpider(Spider):
+ name = "test"
+ start_urls = ["data:,"]
+ custom_settings = {
+ "ITEM_PIPELINES": {SlowDown: 0},
+ "RESPONSE_MAX_ACTIVE_SIZE": 1,
+ }
+
+ async def parse(self, response):
+ assert self.crawler.engine
+ response = await self.crawler.engine.download_async(Request("data:,a"))
+ yield {"response": response}
+
+ crawler = get_crawler(TestSpider)
+ self.caplog.clear()
+ with self.caplog.at_level("INFO"):
+ await maybe_deferred_to_future(crawler.crawl())
+
+ assert _count_backout_logs(self.caplog) == 1
+
+ expected_stats = {
+ "request_backout_seconds/response_max_active_size": gt(0),
+ "request_backout_seconds/total": gt(0),
+ }
+ assert _backout_stats(crawler) == expected_stats
diff --git a/tox.ini b/tox.ini
index 0feb4b3d5..6a0dc4888 100644
--- a/tox.ini
+++ b/tox.ini
@@ -140,7 +140,7 @@ deps =
# pytest 8.4.1 adds support for Twisted 25.5.0 but drops support for Twisted < 24.10.0
pytest==8.4.0
Protego==0.2.0
- Twisted==21.7.0
+ Twisted==23.8.0
aiodns==3.3.0
aioftp==0.26.1
aiohttp==3.13.3
@@ -207,7 +207,7 @@ basepython = {[min]basepython}
deps =
{[min]deps}
Pillow==9.2.0
- Twisted[http2]==21.7.0
+ Twisted[http2]==23.8.0
python-socks[asyncio]==2.6.0
siosocks==0.2.0 # SOCKS server for tests
boto3==1.35.2