Scrapy, a fast high-level web crawling & scraping framework for Python.
Add an FTP download handler for reactorless mode (#8213)
Adrian53b34f729334
diff --git a/docs/intro/install.rst b/docs/intro/install.rstindex 2efdf9f10..faeb4f927 100644--- a/docs/intro/install.rst+++ b/docs/intro/install.rst@@ -115,6 +115,8 @@ The following extras are available:- Reading the system clipboard with :command:`genrequest`* - ``color``- :setting:`LOG_COLOR`+ * - ``ftp-socks``+ - SOCKS proxy support in :ref:`aioftp-handler`* - ``gcs``- :ref:`Google Cloud Storage <topics-feed-storage-gcs>` for:ref:`feed exports <topics-feed-exports>` anddiff --git a/docs/news.rst b/docs/news.rstindex c133a5f53..fd007243b 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -69,6 +69,16 @@ Backward-incompatible changeseffect.(:gh:`8181`)+- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now+ enforces :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE`, also+ when using :reqmeta:`ftp_local_filename`, so downloads bigger than 1 GiB+ now fail by default. To download bigger files, raise the limit to a size+ that fits them, with :setting:`DOWNLOAD_MAXSIZE` or with the+ :reqmeta:`download_maxsize` request meta key for specific requests. You can+ also set the limit to ``0`` to lift it altogether.++ (:gh:`8213`)+.. _release-2.19.0:Scrapy 2.19.0 (2026-09-10)diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex b3cd35f10..643e274ec 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -189,7 +189,8 @@ in future Scrapy versions. The following features are not available:is likely the biggest difference; Scrapy provides an HTTP(S) download handlerthat doesn't require a reactor and will be used instead of it::class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`)-* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`+* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` (replaced+ by :class:`~scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler`)* :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`* :ref:`topics-telnetconsole`* :class:`~scrapy.crawler.CrawlerRunner` and@@ -215,7 +216,8 @@ the defaults of some other settings:* :setting:`TELNETCONSOLE_ENABLED` is set to ``False``.* The ``"http"`` and ``"https"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` areset to ``"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"``.-* The ``"ftp"`` key in :setting:`DOWNLOAD_HANDLERS_BASE` is set to ``None``.+* The ``"ftp"`` and ``"ftps"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` are+ set to ``"scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler"``.Thus, :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`is used by default for making HTTP(S) requests. Please refer to itsdiff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex 19e5de2e3..ab87a0ebb 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -450,6 +450,51 @@ Whether to enable HTTP/2 support in this handler.Built-in non-HTTP download handlers reference=============================================+.. _aioftp-handler:++AioftpDownloadHandler+---------------------++.. versionadded:: VERSION++.. autoclass:: scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler++| Supported schemes: ``ftp``, ``ftps``.+| :ref:`Lazy <lazy-download-handlers>`: yes.+| :ref:`Requires asyncio support <using-asyncio>`: yes.+| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.++This handler supports ``ftp://host/path`` FTP URIs and ``ftps://host/path``+FTPS URIs. FTPS uses `implicit TLS`_ on port 990 and `explicit TLS`_ on any+other port.++.. _implicit TLS: https://en.wikipedia.org/wiki/FTPS#Implicit+.. _explicit TLS: https://en.wikipedia.org/wiki/FTPS#Explicit++It's implemented using the aioftp_ library.++.. _aioftp: https://aioftp.rtfd.io/++A path ending in ``/`` is listed instead of downloaded. The response body is+the plain-text output of the FTP ``NLST`` command, one file or directory name+per line.++It supports SOCKS4 and SOCKS5 :reqmeta:`proxies <proxy>` if you install the+:ref:`ftp-socks <extras>` extra.++.. warning::++ This handler is experimental, and not yet recommended for production+ environments. Future Scrapy versions may introduce related changes without+ a deprecation period or warning or even remove it altogether.++Limitations:++- Only passive mode is supported: the handler cannot be used if+ :setting:`FTP_PASSIVE_MODE` is ``False``, and requests that set the+ :reqmeta:`ftp_passive` request meta key to ``False`` are downloaded in+ passive mode, with an error logged.+DataURIDownloadHandler----------------------diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex b96ac5f4e..204e0d338 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1122,7 +1122,8 @@ Default:"http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler","https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler","s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",- "ftp": None,+ "ftp": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",+ "ftps": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",}(when :setting:`TWISTED_REACTOR_ENABLED` is ``False``)@@ -1256,7 +1257,8 @@ Use ``0`` to disable this limit... note::This limit can be set per-request using the :reqmeta:`download_warnsize`- :attr:`.Request.meta` key.+ :attr:`.Request.meta` key. Requests that set :reqmeta:`download_maxsize`+ but not :reqmeta:`download_warnsize` log no warning... note::diff --git a/pyproject.toml b/pyproject.tomlindex ad402b3ec..602ee6041 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -8,6 +8,7 @@ dynamic = ["version"]description = "A high-level Web Crawling and Web Scraping framework"dependencies = ["Twisted>=21.7.0",+ "aioftp>=0.26.1","aiohttp>=3.13.3","charset-normalizer>=3.4.0","cryptography>=41.0.5",@@ -72,6 +73,7 @@ Tracker = "https://github.com/scrapy/scrapy/issues"bpython = ["bpython>=0.7.1"]clipboard = ["pyperclip>=1.8.0"]color = ["colorlog>=6.6.0"]+ftp-socks = ["python-socks[asyncio]>=2.6.0"]gcs = ["google-cloud-storage>=1.30.0"]httpx = ["httpx2[http2,socks]>=2.0.0"]images = ["Pillow>=9.2.0"]@@ -153,6 +155,7 @@ module = ["pyperclip","pytest_twisted","robotexclusionrulesparser",+ "siosocks.*","zope.interface.*",]ignore_missing_imports = truediff --git a/scrapy/core/_http2/stream.py b/scrapy/core/_http2/stream.pyindex 8dca37e2e..ab99d68c8 100644--- a/scrapy/core/_http2/stream.py+++ b/scrapy/core/_http2/stream.py@@ -19,6 +19,7 @@ from scrapy.http.headers import Headersfrom scrapy.utils._download_handlers import (check_stop_download,get_maxsize_msg,+ get_warnsize,get_warnsize_msg,make_response,)@@ -126,9 +127,7 @@ class Stream:self._download_maxsize = self._request.meta.get("download_maxsize", download_maxsize)- self._download_warnsize = self._request.meta.get(- "download_warnsize", download_warnsize- )+ self._download_warnsize = get_warnsize(self._request.meta, download_warnsize)# Metadata of an HTTP/2 connection stream# initialized when stream is instantiated@@ -190,7 +189,7 @@ class Stream:self._response["headers"].get(b"Content-Length", -1))return (- self._download_warnsize+ bool(self._download_warnsize)and (self._response["flow_controlled_size"] > self._download_warnsizeor content_length_header > self._download_warnsizediff --git a/scrapy/core/downloader/handlers/_aioftp.py b/scrapy/core/downloader/handlers/_aioftp.pynew file mode 100644index 000000000..e365123fc--- /dev/null+++ b/scrapy/core/downloader/handlers/_aioftp.py@@ -0,0 +1,274 @@+from __future__ import annotations++import asyncio+import ipaddress+import logging+import socket+import ssl+from contextlib import asynccontextmanager+from pathlib import Path+from typing import TYPE_CHECKING, Any, BinaryIO, ClassVar, NamedTuple+from urllib.parse import unquote, urlsplit++import aioftp+from aioftp.common import SSLSessionBoundContext++from scrapy.exceptions import (+ CannotResolveHostError,+ DownloadConnectionRefusedError,+ DownloadFailedError,+ DownloadTimeoutError,+ NotConfigured,+)+from scrapy.http import Headers+from scrapy.utils._ssl import _log_sslobj_debug_info, _make_ssl_context+from scrapy.utils.httpobj import urlparse_cached++from ._base_streaming import BaseStreamingDownloadHandler, _BaseResponseArgs++if TYPE_CHECKING:+ from collections.abc import AsyncGenerator, AsyncIterator++ from scrapy.crawler import Crawler+ from scrapy.http import Response+ from scrapy.http.request import Request+++logger = logging.getLogger(__name__)++_TIMEOUT_EXCEPTIONS: tuple[type[Exception], ...] = (TimeoutError,)+_CONNECTION_REFUSED_EXCEPTIONS: tuple[type[Exception], ...] = (ConnectionRefusedError,)+_DOWNLOAD_FAILED_EXCEPTIONS: tuple[type[Exception], ...] = (+ aioftp.AIOFTPException,+ ConnectionResetError,+)+_HAS_SOCKS = False+try:+ from python_socks import (+ ProxyConnectionError,+ ProxyError,+ ProxyTimeoutError,+ ProxyType,+ )+ from python_socks.async_.asyncio import Proxy++ _HAS_SOCKS = True+ _TIMEOUT_EXCEPTIONS += (ProxyTimeoutError,)+ _CONNECTION_REFUSED_EXCEPTIONS += (ProxyConnectionError,)+ _DOWNLOAD_FAILED_EXCEPTIONS += (ProxyError,)+except ImportError: # pragma: no cover+ pass+++class _ProxyClient(aioftp.Client):+ """An :class:`aioftp.Client` that opens its connections through+ *proxy*."""++ def __init__(self, proxy: Proxy, **kwargs: Any):+ super().__init__(**kwargs)+ self._proxy = proxy++ async def _open_connection(+ self, host: str, port: int+ ) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:+ ssl_context = self.ssl+ if self._stream is not None and (+ ssl_object := self._stream.writer.get_extra_info("ssl_object")+ ):+ # data connections reuse the TLS session of the control connection+ ssl_context = SSLSessionBoundContext(+ ssl.PROTOCOL_TLS_CLIENT,+ context=ssl_object.context,+ session=ssl_object.session,+ )+ sock = await self._proxy.connect(host, port, timeout=self.connection_timeout)+ return await asyncio.open_connection(+ sock=sock,+ ssl=ssl_context or None,+ server_hostname=host if ssl_context else None,+ )+++class _AioftpResponse(NamedTuple):+ status: int # mapped to common HTTP status codes+ control: aioftp.ThrottleStreamIO+ data: aioftp.DataConnectionThrottleStreamIO | None+++class AioftpDownloadHandler(BaseStreamingDownloadHandler[_AioftpResponse]):+ lazy = True+ experimental: ClassVar[bool] = True++ def __init__(self, crawler: Crawler):+ super().__init__(crawler)+ # aioftp only supports passive mode+ if not crawler.settings.getbool("FTP_PASSIVE_MODE"):+ raise NotConfigured(+ f"{type(self).__name__} does not support disabling FTP_PASSIVE_MODE."+ )+ self._user: str = crawler.settings["FTP_USER"]+ self._password: str = crawler.settings["FTP_PASSWORD"]+ self._ssl_context: ssl.SSLContext = _make_ssl_context(crawler.settings)++ @asynccontextmanager+ async def _make_request(+ self,+ request: Request,+ timeout: float, # noqa: ASYNC109+ ) -> AsyncGenerator[_AioftpResponse]:+ url = urlparse_cached(request)+ assert url.hostname+ if not request.meta.get("ftp_passive", True):+ logger.error(+ f"{type(self).__name__} does not support active mode, so the"+ f" 'ftp_passive' request meta key will be ignored for {request}."+ )++ implicit_tls = url.scheme == "ftps" and url.port == 990+ client_kwargs: dict[str, Any] = {+ "socket_timeout": timeout,+ "connection_timeout": timeout,+ "ssl": self._ssl_context if implicit_tls else None,+ }+ proxy = self._get_proxy(request)+ client = (+ _ProxyClient(proxy, **client_kwargs)+ if proxy+ else aioftp.Client(**client_kwargs)+ )+ try:+ await client.connect(url.hostname, url.port or 21)+ if url.scheme == "ftps" and not implicit_tls:+ await client.upgrade_to_tls(self._ssl_context)+ await client.login(+ request.meta.get("ftp_user", self._user),+ request.meta.get("ftp_password", self._password),+ )+ reader, writer = await client.get_passive_connection()+ data = aioftp.DataConnectionThrottleStreamIO(+ client, reader, writer, throttles={}, timeout=timeout+ )+ try:+ path = unquote(url.path)+ command = "NLST" if not path or path.endswith("/") else "RETR"+ await client.command(f"{command} {path}".rstrip(), "1xx")+ except aioftp.StatusCodeError as e:+ data.close()+ status = 404 if e.received_codes[-1].matches("550") else 503+ yield _AioftpResponse(status, client.stream, None)+ else:+ try:+ yield _AioftpResponse(200, client.stream, data)+ finally:+ data.close()+ except _TIMEOUT_EXCEPTIONS as e:+ raise DownloadTimeoutError(+ f"Getting {request.url} took longer than {timeout} seconds."+ ) from e+ except _CONNECTION_REFUSED_EXCEPTIONS as e:+ raise DownloadConnectionRefusedError(str(e)) from e+ except socket.gaierror as e:+ raise CannotResolveHostError(str(e)) from e+ except _DOWNLOAD_FAILED_EXCEPTIONS as e:+ raise DownloadFailedError(str(e)) from e+ finally:+ client.close()++ def _get_proxy(self, request: Request) -> Proxy | None:+ proxy_url = self._extract_proxy_url_with_creds(request)+ if not proxy_url:+ return None+ parts = urlsplit(proxy_url)+ if parts.scheme not in ("socks4", "socks5"):+ raise NotImplementedError(+ f"{type(self).__name__} only supports SOCKS proxies, got {proxy_url!r}."+ )+ if not _HAS_SOCKS: # pragma: no cover+ raise ValueError(+ f"SOCKS proxy support in {type(self).__name__} requires the"+ f" 'ftp-socks' extra to be installed."+ )+ assert parts.hostname+ return Proxy(+ ProxyType.SOCKS4 if parts.scheme == "socks4" else ProxyType.SOCKS5,+ parts.hostname,+ parts.port or 1080,+ unquote(parts.username) if parts.username else None,+ unquote(parts.password) if parts.password else None,+ )++ @staticmethod+ def _local_filename(response: _AioftpResponse, request: Request) -> bytes | None:+ if response.data is None:+ return None+ return request.meta.get("ftp_local_filename")++ def _open_body_buffer(+ self, response: _AioftpResponse, request: Request+ ) -> BinaryIO:+ if local_filename := self._local_filename(response, request):+ return Path(local_filename.decode()).open("wb")+ return super()._open_body_buffer(response, request)++ def _get_body(+ self, buffer: BinaryIO, response: _AioftpResponse, request: Request+ ) -> bytes:+ if local_filename := self._local_filename(response, request):+ return local_filename+ return super()._get_body(buffer, response, request)++ async def _read_response(+ self, response: _AioftpResponse, request: Request+ ) -> Response:+ result = await super()._read_response(response, request)+ if response.data is not None:+ self._add_ftp_headers(result, response, request)+ return result++ def _add_ftp_headers(+ self, result: Response, response: _AioftpResponse, request: Request+ ) -> None:+ if local_filename := self._local_filename(response, request):+ size = Path(local_filename.decode()).stat().st_size+ else:+ size = len(result.body)+ result.headers.update({"Local Filename": local_filename or b"", "Size": size})++ @staticmethod+ def _extract_headers(response: _AioftpResponse) -> Headers:+ return Headers()++ @staticmethod+ def _build_base_response_args(+ response: _AioftpResponse,+ request: Request,+ headers: Headers,+ ) -> _BaseResponseArgs:+ peername = response.control.writer.get_extra_info("peername")+ ssl_object = response.control.writer.get_extra_info("ssl_object")+ cert = ssl_object.getpeercert(binary_form=True) if ssl_object else None+ return {+ "status": response.status,+ "url": request.url,+ "certificate": cert,+ "protocol": "FTP",+ "ip_address": ipaddress.ip_address(peername[0]),+ "headers": headers,+ }++ @staticmethod+ async def _iter_body_chunks(response: _AioftpResponse) -> AsyncIterator[bytes]:+ if response.data is None:+ return+ async for chunk in response.data.iter_by_block():+ yield chunk+ # raises StatusCodeError if the server reports a failed transfer+ await response.data.finish()++ @staticmethod+ def _is_dataloss_exception(exc: Exception) -> bool:+ return isinstance(exc, aioftp.StatusCodeError)++ def _log_tls_info(self, response: _AioftpResponse, request: Request) -> None:+ if ssl_object := response.control.writer.get_extra_info("ssl_object"):+ _log_sslobj_debug_info(ssl_object)diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.pyindex 596942169..1888e7a26 100644--- a/scrapy/core/downloader/handlers/_base_streaming.py+++ b/scrapy/core/downloader/handlers/_base_streaming.py@@ -8,6 +8,7 @@ from io import BytesIOfrom typing import (TYPE_CHECKING,Any,+ BinaryIO,ClassVar,Generic,NoReturn,@@ -27,6 +28,7 @@ from scrapy.utils._download_handlers import (check_stop_download,get_dataloss_msg,get_maxsize_msg,+ get_warnsize,get_warnsize_msg,make_response,normalize_bind_address,@@ -140,6 +142,17 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respondef _log_tls_info(self, response: _ResponseT, request: Request) -> None:"""Log TLS connection details, if possible."""+ def _open_body_buffer(self, response: _ResponseT, request: Request) -> BinaryIO:+ """Return a file object to write the response body into."""+ return BytesIO()++ def _get_body(+ self, buffer: BinaryIO, response: _ResponseT, request: Request+ ) -> bytes:+ """Return the response body given the *buffer* it was written into."""+ assert isinstance(buffer, BytesIO)+ return buffer.getvalue()+async def download_request(self, request: Request) -> Response:if not self.supports_proxies and request.meta.get("proxy"):raise NotImplementedError(f"{type(self).__name__} doesn't support proxies.")@@ -160,7 +173,7 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Responasync def _read_response(self, response: _ResponseT, request: Request) -> Response:maxsize: int = request.meta.get("download_maxsize", self._default_maxsize)- warnsize: int = request.meta.get("download_warnsize", self._default_warnsize)+ warnsize = get_warnsize(request.meta, self._default_warnsize)headers = self._extract_headers(response)content_length = headers.get("Content-Length")@@ -194,56 +207,56 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Responstop_download=stop_download,)- response_body = BytesIO()- bytes_received = 0- try:- async for chunk in self._iter_body_chunks(response):- response_body.write(chunk)- bytes_received += len(chunk)+ with self._open_body_buffer(response, request) as response_body:+ bytes_received = 0+ try:+ async for chunk in self._iter_body_chunks(response):+ response_body.write(chunk)+ bytes_received += len(chunk)++ if stop_download := check_stop_download(+ signals.bytes_received, self.crawler, request, data=chunk+ ):+ return make_response(+ **make_response_base_args,+ body=self._get_body(response_body, response, request),+ stop_download=stop_download,+ )++ if maxsize and bytes_received > maxsize:+ response_body.truncate(0)+ self._cancel_maxsize(+ bytes_received, maxsize, request, expected=False+ )- if stop_download := check_stop_download(- signals.bytes_received, self.crawler, request, data=chunk- ):+ if warnsize and bytes_received > warnsize and not reached_warnsize:+ reached_warnsize = True+ logger.warning(+ get_warnsize_msg(+ bytes_received, warnsize, request, expected=False+ )+ )+ except Exception as e:+ if not self._is_dataloss_exception(e):+ raise+ fail_on_dataloss: bool = request.meta.get(+ "download_fail_on_dataloss", self._fail_on_dataloss+ )+ if not fail_on_dataloss:return make_response(**make_response_base_args,- body=response_body.getvalue(),- stop_download=stop_download,- )-- if maxsize and bytes_received > maxsize:- response_body.truncate(0)- self._cancel_maxsize(- bytes_received, maxsize, request, expected=False+ body=self._get_body(response_body, response, request),+ flags=["dataloss"],)+ if not self._fail_on_dataloss_warned:+ logger.warning(get_dataloss_msg(request.url))+ self._fail_on_dataloss_warned = True+ raise ResponseDataLossError(str(e)) from e- if warnsize and bytes_received > warnsize and not reached_warnsize:- reached_warnsize = True- logger.warning(- get_warnsize_msg(- bytes_received, warnsize, request, expected=False- )- )- except Exception as e:- if not self._is_dataloss_exception(e):- raise- fail_on_dataloss: bool = request.meta.get(- "download_fail_on_dataloss", self._fail_on_dataloss+ return make_response(+ **make_response_base_args,+ body=self._get_body(response_body, response, request),)- if not fail_on_dataloss:- return make_response(- **make_response_base_args,- body=response_body.getvalue(),- flags=["dataloss"],- )- if not self._fail_on_dataloss_warned:- logger.warning(get_dataloss_msg(request.url))- self._fail_on_dataloss_warned = True- raise ResponseDataLossError(str(e)) from e-- return make_response(- **make_response_base_args,- body=response_body.getvalue(),- )@staticmethoddef _request_headers(request: Request) -> Headers:diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.pyindex 327193a29..e3b8f6331 100644--- a/scrapy/core/downloader/handlers/ftp.py+++ b/scrapy/core/downloader/handlers/ftp.py@@ -30,6 +30,7 @@ In case of status 200 request, response.headers will come with two keys:from __future__ import annotations+import loggingimport refrom io import BytesIOfrom pathlib import Path@@ -39,9 +40,14 @@ from urllib.parse import unquotefrom twisted.internet.protocol import ClientCreator, Protocolfrom scrapy.core.downloader.handlers.base import BaseDownloadHandler-from scrapy.exceptions import DownloadFailedError, NotConfigured+from scrapy.exceptions import DownloadCancelledError, DownloadFailedError, NotConfiguredfrom scrapy.http import Responsefrom scrapy.responsetypes import responsetypes+from scrapy.utils._download_handlers import (+ get_maxsize_msg,+ get_warnsize,+ get_warnsize_msg,+)from scrapy.utils.defer import maybe_deferred_to_futurefrom scrapy.utils.httpobj import urlparse_cached@@ -52,17 +58,47 @@ if TYPE_CHECKING:from scrapy.crawler import Crawler+logger = logging.getLogger(__name__)++class ReceivedDataProtocol(Protocol):- def __init__(self, filename: bytes | None = None):+ def __init__(+ self,+ filename: bytes | None = None,+ *,+ request: Request | None = None,+ maxsize: int = 0,+ warnsize: int = 0,+ ):self.__filename: bytes | None = filenameself.body: BinaryIO = (Path(filename.decode()).open("wb") if filename else BytesIO())self.size: int = 0+ self._request: Request | None = request+ self._maxsize: int = maxsize+ self._warnsize: int = warnsize+ self._exceeded_maxsize: bool = False+ self._reached_warnsize: bool = Falsedef dataReceived(self, data: bytes) -> None:self.body.write(data)self.size += len(data)+ if self._maxsize and self.size > self._maxsize:+ self._exceeded_maxsize = True+ self.body.truncate(0)+ assert self.transport+ self.transport.loseConnection()+ elif (+ self._warnsize and self.size > self._warnsize and not self._reached_warnsize+ ):+ self._reached_warnsize = True+ assert self._request is not None+ logger.warning(+ get_warnsize_msg(+ self.size, self._warnsize, self._request, expected=False+ )+ )@propertydef filename(self) -> bytes | None:@@ -91,6 +127,8 @@ class FTPDownloadHandler(BaseDownloadHandler):self.default_user = crawler.settings["FTP_USER"]self.default_password = crawler.settings["FTP_PASSWORD"]self.passive_mode = crawler.settings["FTP_PASSIVE_MODE"]+ self._default_maxsize: int = crawler.settings.getint("DOWNLOAD_MAXSIZE")+ self._default_warnsize: int = crawler.settings.getint("DOWNLOAD_WARNSIZE")async def download_request(self, request: Request) -> Response:from twisted.internet import reactor@@ -110,26 +148,41 @@ class FTPDownloadHandler(BaseDownloadHandler):)filepath = unquote(parsed_url.path)is_listing = filepath == "" or filepath.endswith("/")- protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))+ maxsize: int = request.meta.get("download_maxsize", self._default_maxsize)+ protocol = ReceivedDataProtocol(+ request.meta.get("ftp_local_filename"),+ request=request,+ maxsize=maxsize,+ warnsize=get_warnsize(request.meta, self._default_warnsize),+ )try:if is_listing:await maybe_deferred_to_future(client.nlst(filepath, protocol))else:await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))except CommandFailed as e:- message = str(e)- # Twisted only raises CommandFailed for a reply whose numeric code- # it has parsed, so the message always carries that code.- m = _CODE_RE.search(message)- assert m- httpcode = self.CODE_MAPPING.get(m.group(), self.CODE_MAPPING["default"])- return Response(url=request.url, status=httpcode, body=message.encode())+ if not protocol._exceeded_maxsize:+ message = str(e)+ # Twisted only raises CommandFailed for a reply whose numeric+ # code it has parsed, so the message always carries that code.+ m = _CODE_RE.search(message)+ assert m+ httpcode = self.CODE_MAPPING.get(+ m.group(), self.CODE_MAPPING["default"]+ )+ return Response(url=request.url, status=httpcode, body=message.encode())except ConnectionLost as e:raise DownloadFailedError(str(e)) from efinally:protocol.close()assert client.transportclient.transport.loseConnection()+ if protocol._exceeded_maxsize:+ warning_msg = get_maxsize_msg(+ protocol.size, maxsize, request, expected=False+ )+ logger.warning(warning_msg)+ raise DownloadCancelledError(warning_msg)headers = {"Local Filename": protocol.filename or b"", "Size": protocol.size}body = protocol.filename or protocol.body.read()respcls = responsetypes.from_args(url=request.url, body=body)diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.pyindex 7146143f1..bcb32a076 100644--- a/scrapy/core/downloader/handlers/http11.py+++ b/scrapy/core/downloader/handlers/http11.py@@ -52,6 +52,7 @@ from scrapy.utils._download_handlers import (check_stop_download,get_dataloss_msg,get_maxsize_msg,+ get_warnsize,get_warnsize_msg,make_response,normalize_bind_address,@@ -585,7 +586,7 @@ class _ScrapyAgent:}maxsize = request.meta.get("download_maxsize", self._maxsize)- warnsize = request.meta.get("download_warnsize", self._warnsize)+ warnsize = get_warnsize(request.meta, self._warnsize)expected_size = (cast("int", txresponse.length)if txresponse.length != UNKNOWN_LENGTHdiff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex fd75fba9a..cb42126a8 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -283,7 +283,10 @@ class Crawler:self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = ("scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler")- self.settings["DOWNLOAD_HANDLERS_BASE"]["ftp"] = None+ for scheme in ("ftp", "ftps"):+ self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = (+ "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler"+ )# Cannot use @deferred_f_from_coro_f because that relies on the reactor# being installed already, which is done within _apply_settings(), insidediff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.pyindex 5ab9eac12..8beee247e 100644--- a/scrapy/downloadermiddlewares/httpcompression.py+++ b/scrapy/downloadermiddlewares/httpcompression.py@@ -22,6 +22,7 @@ from scrapy.utils._compression import (_unzstd,gunzip,)+from scrapy.utils._download_handlers import get_warnsizefrom scrapy.utils.asyncio import run_in_threadfrom scrapy.utils.decorators import _warn_spider_argfrom scrapy.utils.deprecate import warn_on_deprecated_spider_attribute@@ -101,7 +102,7 @@ class HttpCompressionMiddleware:content_encoding = response.headers.getlist("Content-Encoding")if content_encoding:max_size = request.meta.get("download_maxsize", self._max_size)- warn_size = request.meta.get("download_warnsize", self._warn_size)+ warn_size = get_warnsize(request.meta, self._warn_size)try:decoded_body, content_encoding = await self._handle_encoding(response.body, content_encoding, max_sizediff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.pyindex 425466a27..0c20c1e1d 100644--- a/scrapy/spiders/sitemap.py+++ b/scrapy/spiders/sitemap.py@@ -246,7 +246,10 @@ class SitemapSpider(Spider):if gzip_magic_number(response):uncompressed_size = len(response.body)max_size = response.meta.get("download_maxsize", self._max_size)- warn_size = response.meta.get("download_warnsize", self._warn_size)+ # circular import+ from scrapy.utils._download_handlers import get_warnsize # noqa: PLC0415++ warn_size = get_warnsize(response.meta, self._warn_size)try:body = gunzip(response.body, max_size=max_size)except _DecompressionMaxSizeExceeded:diff --git a/scrapy/utils/_download_handlers.py b/scrapy/utils/_download_handlers.pyindex 23b715ac6..57330bd2b 100644--- a/scrapy/utils/_download_handlers.py+++ b/scrapy/utils/_download_handlers.py@@ -129,6 +129,20 @@ def get_maxsize_msg(size: int, limit: int, request: Request, *, expected: bool))+def get_warnsize(meta: dict[str, Any], default: int) -> int:+ """Return the download warn size for a request with the given *meta*.++ A request-specific ``download_maxsize`` disables the *default* warn size,+ since a user who sets a request-specific size limit expects that size.+ """+ if "download_warnsize" in meta:+ warnsize: int = meta["download_warnsize"]+ return warnsize+ if "download_maxsize" in meta:+ return 0+ return default++def get_warnsize_msg(size: int, limit: int, request: Request, *, expected: bool) -> str:prefix = "Expected to receive" if expected else "Received"return (diff --git a/scrapy/utils/test.py b/scrapy/utils/test.pyindex d461056e5..8f11e70aa 100644--- a/scrapy/utils/test.py+++ b/scrapy/utils/test.py@@ -50,7 +50,8 @@ def get_reactor_settings() -> dict[str, Any]:# last one from them.settings["TWISTED_REACTOR_ENABLED"] = Falsesettings["DOWNLOAD_HANDLERS"] = {- "ftp": None,+ "ftp": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",+ "ftps": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler","http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler","https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",}diff --git a/tests/mockserver/ftp.py b/tests/mockserver/ftp.pyindex d09bf6115..2398466c6 100644--- a/tests/mockserver/ftp.py+++ b/tests/mockserver/ftp.py@@ -7,6 +7,7 @@ from pathlib import Pathfrom shutil import rmtreefrom subprocess import PIPE, Popenfrom tempfile import mkdtemp+from threading import Threadfrom typing import TYPE_CHECKING, Selffrom pyftpdlib.authorizers import DummyAuthorizer@@ -29,6 +30,7 @@ class MockFTPServer:"""proc: Popen[str]+ _stderr_reader: Threadport: intpath: Path@@ -56,6 +58,9 @@ class MockFTPServer:raise RuntimeError("The FTP server failed to start or the output is unrecognized")+ # Keep the pipe from filling up, which would block the server.+ self._stderr_reader = Thread(target=self.proc.stderr.read, daemon=True)+ self._stderr_reader.start()return selfdef __exit__(@@ -66,6 +71,7 @@ class MockFTPServer:) -> None:rmtree(str(self.path))self.proc.kill()+ self._stderr_reader.join()self.proc.communicate()def url(self, path: str) -> str:diff --git a/tests/test_downloader_handler_aioftp.py b/tests/test_downloader_handler_aioftp.pynew file mode 100644index 000000000..bf98920f7--- /dev/null+++ b/tests/test_downloader_handler_aioftp.py@@ -0,0 +1,409 @@+from __future__ import annotations++import asyncio+import logging+from contextlib import asynccontextmanager+from ipaddress import ip_address+from typing import TYPE_CHECKING, Any++import pytest+from w3lib.http import basic_auth_header++from scrapy.core.downloader.handlers._aioftp import _HAS_SOCKS, AioftpDownloadHandler+from scrapy.exceptions import (+ CannotResolveHostError,+ DownloadCancelledError,+ DownloadConnectionRefusedError,+ DownloadFailedError,+ DownloadTimeoutError,+ NotConfigured,+ ResponseDataLossError,+)+from scrapy.http import HtmlResponse, Request, TextResponse+from scrapy.utils.misc import build_from_crawler+from scrapy.utils.test import get_crawler+from tests.mockserver.ftp import MockFTPServer+from tests.utils.decorators import coroutine_test++try:+ from siosocks.io.asyncio import socks_server_handler+except ImportError:+ socks_server_handler = None++if TYPE_CHECKING:+ from collections.abc import AsyncGenerator, Callable, Generator+ from pathlib import Path++ from scrapy.http import Response++pytestmark = pytest.mark.only_asyncio++FILES = {+ "file.txt": b"I have the power!",+ "file with spaces.txt": b"Moooooooooo power!",+ "html-file-without-extension": b"<!DOCTYPE html>\n<title>.</title>",+}+++def _write_files(server: MockFTPServer) -> MockFTPServer:+ for name, content in FILES.items():+ (server.path / name).write_bytes(content)+ return server++[email protected](scope="module")+def ftp_server() -> Generator[MockFTPServer]:+ with MockFTPServer() as server:+ yield _write_files(server)++[email protected](scope="module")+def ftps_server() -> Generator[MockFTPServer]:+ with MockFTPServer(tls=True) as server:+ yield _write_files(server)+++async def _download(+ request: Request, settings: dict[str, Any] | None = None+) -> Response:+ crawler = get_crawler(settings_dict=settings)+ dh = build_from_crawler(AioftpDownloadHandler, crawler)+ try:+ return await dh.download_request(request)+ finally:+ await dh.close()+++ ("path", "response_class"),+ [+ ("file.txt", TextResponse),+ ("file with spaces.txt", TextResponse),+ ("html-file-without-extension", HtmlResponse),+ ],+)+@coroutine_test+async def test_download(+ ftp_server: MockFTPServer, path: str, response_class: type[Response]+) -> None:+ response = await _download(Request(ftp_server.url(path)))+ assert type(response) is response_class+ assert response.status == 200+ assert response.body == FILES[path]+ assert response.headers == {+ b"Local Filename": [b""],+ b"Size": [str(len(FILES[path])).encode()],+ }+ assert response.protocol == "FTP"+ assert response.ip_address == ip_address("127.0.0.1")+ assert response.certificate is None+++@coroutine_test+async def test_download_ftps(+ ftps_server: MockFTPServer, caplog: pytest.LogCaptureFixture+) -> None:+ settings = {"DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING": True}+ with caplog.at_level(logging.DEBUG):+ response = await _download(Request(ftps_server.url("file.txt")), settings)+ assert response.body == FILES["file.txt"]+ assert isinstance(response.certificate, bytes)+ assert "SSL connection to 127.0.0.1 using protocol" in caplog.text+++@coroutine_test+async def test_tls_verbose_logging_without_tls(ftp_server: MockFTPServer) -> None:+ settings = {"DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING": True}+ response = await _download(Request(ftp_server.url("file.txt")), settings)+ assert response.body == FILES["file.txt"]+++@coroutine_test+async def test_nonexistent(ftp_server: MockFTPServer) -> None:+ response = await _download(Request(ftp_server.url("nonexistent.txt")))+ assert response.status == 404+ assert response.body == b""++[email protected]("path", ["", "/"])+@coroutine_test+async def test_list(ftp_server: MockFTPServer, path: str) -> None:+ url = f"ftp://{ftp_server.host}:{ftp_server.port}{path}"+ response = await _download(Request(url))+ assert isinstance(response, TextResponse)+ assert response.status == 200+ assert set(response.text.splitlines()) == set(FILES)+++@coroutine_test+async def test_list_nonexistent(ftp_server: MockFTPServer) -> None:+ response = await _download(Request(ftp_server.url("nonexistent/")))+ assert response.status == 404+ assert response.body == b""+++@coroutine_test+async def test_invalid_credentials(ftp_server: MockFTPServer) -> None:+ request = Request(ftp_server.url("file.txt"), meta={"ftp_user": "invalid"})+ with pytest.raises(DownloadFailedError):+ await _download(request)++[email protected]("local", [False, True])+@coroutine_test+async def test_maxsize(ftp_server: MockFTPServer, tmp_path: Path, local: bool) -> None:+ meta: dict[str, Any] = {"download_maxsize": 10}+ if local:+ meta["ftp_local_filename"] = str(tmp_path / "file.txt").encode()+ request = Request(ftp_server.url("file.txt"), meta=meta)+ with pytest.raises(DownloadCancelledError):+ await _download(request)+++@coroutine_test+async def test_maxsize_disabled(ftp_server: MockFTPServer) -> None:+ request = Request(ftp_server.url("file.txt"), meta={"download_maxsize": 0})+ response = await _download(request, {"DOWNLOAD_MAXSIZE": 10})+ assert response.body == FILES["file.txt"]+++@coroutine_test+async def test_warnsize(+ ftp_server: MockFTPServer, caplog: pytest.LogCaptureFixture+) -> None:+ request = Request(ftp_server.url("file.txt"), meta={"download_warnsize": 10})+ response = await _download(request)+ assert response.body == FILES["file.txt"]+ assert "which is larger than download warn size (10)" in caplog.text+++@coroutine_test+async def test_active_mode_meta(+ ftp_server: MockFTPServer, caplog: pytest.LogCaptureFixture+) -> None:+ request = Request(ftp_server.url("file.txt"), meta={"ftp_passive": False})+ response = await _download(request)+ assert response.body == FILES["file.txt"]+ assert "does not support active mode" in caplog.text+++@coroutine_test+async def test_active_mode() -> None:+ crawler = get_crawler(settings_dict={"FTP_PASSIVE_MODE": False})+ with pytest.raises(NotConfigured):+ build_from_crawler(AioftpDownloadHandler, crawler)+++@coroutine_test+async def test_connection_refused() -> None:+ server = await asyncio.start_server(lambda r, w: None, "127.0.0.1", 0)+ port = server.sockets[0].getsockname()[1]+ server.close()+ await server.wait_closed()+ with pytest.raises(DownloadConnectionRefusedError):+ await _download(Request(f"ftp://127.0.0.1:{port}/file.txt"))+++@coroutine_test+async def test_cannot_resolve_host() -> None:+ with pytest.raises(CannotResolveHostError):+ await _download(Request("ftp://nonexistent.invalid/file.txt"))+++@coroutine_test+async def test_timeout() -> None:+ async def never_greet(+ reader: asyncio.StreamReader, writer: asyncio.StreamWriter+ ) -> None:+ await reader.read()+ writer.close()++ async with _serve(never_greet) as url:+ request = Request(url, meta={"download_timeout": 0.1})+ with pytest.raises(DownloadTimeoutError):+ await _download(request)+++@coroutine_test+async def test_connection_lost() -> None:+ async with _serve(lambda r, w: w.close()) as url:+ with pytest.raises(DownloadFailedError):+ await _download(Request(url))+++def _fake_ftp(+ retr_reply: bytes, data: bytes = b""+) -> Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any]:+ """Return a minimal FTP server connection handler that sends *data*, if+ any, and then answers RETR with *retr_reply*."""++ async def handle(+ reader: asyncio.StreamReader, writer: asyncio.StreamWriter+ ) -> None:+ data_writer: asyncio.Future[asyncio.StreamWriter] = (+ asyncio.get_running_loop().create_future()+ )++ def on_data_connection(+ _: asyncio.StreamReader, w: asyncio.StreamWriter+ ) -> None:+ data_writer.set_result(w)++ writer.write(b"220 Ready\r\n")+ while line := await reader.readline():+ command = line.split()[0].upper()+ if command == b"USER":+ writer.write(b"230 Logged in\r\n")+ elif command == b"TYPE":+ writer.write(b"200 OK\r\n")+ elif command == b"EPSV":+ data_server = await asyncio.start_server(+ on_data_connection, "127.0.0.1", 0+ )+ port = data_server.sockets[0].getsockname()[1]+ writer.write(f"229 Entering passive mode (|||{port}|)\r\n".encode())+ elif command == b"RETR":+ w = await data_writer+ if data:+ writer.write(b"150 Sending\r\n")+ w.write(data)+ w.close()+ data_server.close()+ writer.write(retr_reply)+ await writer.drain()+ writer.close()++ return handle+++@asynccontextmanager+async def _serve(+ handler: Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any],+) -> AsyncGenerator[str]:+ server = await asyncio.start_server(handler, "127.0.0.1", 0)+ port = server.sockets[0].getsockname()[1]+ try:+ yield f"ftp://127.0.0.1:{port}/file.txt"+ finally:+ server.close()+++@coroutine_test+async def test_retr_failure() -> None:+ async with _serve(_fake_ftp(b"450 Busy\r\n")) as url:+ response = await _download(Request(url))+ assert response.status == 503+++_TRUNCATED = _fake_ftp(b"426 Transfer aborted\r\n", data=b"partial")+++@coroutine_test+async def test_dataloss() -> None:+ async with _serve(_TRUNCATED) as url:+ with pytest.raises(ResponseDataLossError):+ await _download(Request(url))+++@coroutine_test+async def test_dataloss_allowed() -> None:+ async with _serve(_TRUNCATED) as url:+ response = await _download(+ Request(url, meta={"download_fail_on_dataloss": False})+ )+ assert response.body == b"partial"+ assert response.flags == ["dataloss"]+++@coroutine_test+async def test_local_filename(ftp_server: MockFTPServer, tmp_path: Path) -> None:+ local_filename = str(tmp_path / "file.txt").encode()+ request = Request(+ ftp_server.url("file.txt"), meta={"ftp_local_filename": local_filename}+ )+ response = await _download(request)+ assert response.body == local_filename+ assert response.headers == {+ b"Local Filename": [local_filename],+ b"Size": [b"17"],+ }+ assert (tmp_path / "file.txt").read_bytes() == FILES["file.txt"]+++@asynccontextmanager+async def _socks_proxy(**kwargs: Any) -> AsyncGenerator[str]:+ server = await asyncio.start_server(+ lambda r, w: socks_server_handler(r, w, **kwargs), "127.0.0.1", 0+ )+ port = server.sockets[0].getsockname()[1]+ try:+ yield f"127.0.0.1:{port}"+ finally:+ server.close()+++requires_socks = pytest.mark.skipif(+ not _HAS_SOCKS or socks_server_handler is None,+ reason="Requires python-socks and siosocks",+)+++@requires_socks[email protected]("scheme", ["socks4", "socks5"])+@coroutine_test+async def test_socks_proxy(ftp_server: MockFTPServer, scheme: str) -> None:+ async with _socks_proxy() as proxy:+ request = Request(+ ftp_server.url("file.txt"), meta={"proxy": f"{scheme}://{proxy}"}+ )+ response = await _download(request)+ assert response.body == FILES["file.txt"]+++@requires_socks+@coroutine_test+async def test_socks_proxy_credentials(ftp_server: MockFTPServer) -> None:+ async with _socks_proxy(+ allowed_versions={5}, username="user", password="pass"+ ) as proxy:+ request = Request(+ ftp_server.url("file.txt"),+ meta={"proxy": f"socks5://{proxy}"},+ headers={"Proxy-Authorization": basic_auth_header("user", "pass")},+ )+ response = await _download(request)+ assert response.body == FILES["file.txt"]+++@requires_socks+@coroutine_test+async def test_socks_proxy_wrong_credentials(ftp_server: MockFTPServer) -> None:+ async with _socks_proxy(+ allowed_versions={5}, username="user", password="pass"+ ) as proxy:+ request = Request(+ ftp_server.url("file.txt"),+ meta={"proxy": f"socks5://{proxy}"},+ headers={"Proxy-Authorization": basic_auth_header("user", "wrong")},+ )+ with pytest.raises(DownloadFailedError):+ await _download(request)+++@requires_socks+@coroutine_test+async def test_socks_proxy_ftps(ftps_server: MockFTPServer) -> None:+ async with _socks_proxy() as proxy:+ request = Request(+ ftps_server.url("file.txt"), meta={"proxy": f"socks5://{proxy}"}+ )+ response = await _download(request)+ assert response.body == FILES["file.txt"]+ assert isinstance(response.certificate, bytes)+++@coroutine_test+async def test_http_proxy(ftp_server: MockFTPServer) -> None:+ request = Request(ftp_server.url("file.txt"), meta={"proxy": "http://127.0.0.1:1"})+ with pytest.raises(NotImplementedError):+ await _download(request)diff --git a/tests/test_downloader_handler_twisted_ftp.py b/tests/test_downloader_handler_twisted_ftp.pyindex 1f00b9497..0d4d42901 100644--- a/tests/test_downloader_handler_twisted_ftp.py+++ b/tests/test_downloader_handler_twisted_ftp.py@@ -14,7 +14,7 @@ from twisted.internet.protocol import Factory, Protocolfrom scrapy import Spiderfrom scrapy.core.downloader.handlers.ftp import FTPDownloadHandlerfrom scrapy.crawler import Crawler-from scrapy.exceptions import DownloadFailedError, NotConfigured+from scrapy.exceptions import DownloadCancelledError, DownloadFailedError, NotConfiguredfrom scrapy.http import HtmlResponse, Request, Responsefrom scrapy.http.response.text import TextResponsefrom scrapy.utils.defer import deferred_f_from_coro_f, maybe_deferred_to_future@@ -39,6 +39,7 @@ class TestFTPBase(ABC):("file.txt", b"I have the power!"),("file with spaces.txt", b"Moooooooooo power!"),("html-file-without-extension", b"<!DOCTYPE html>\n<title>.</title>"),+ ("big.bin", b"x" * 2**20),)@abstractmethod@@ -144,6 +145,50 @@ class TestFTPBase(ABC):assert local_path.read_bytes() == b"I have the power!"local_path.unlink()+ @pytest.mark.parametrize("filename", ["file.txt", "big.bin"])+ @pytest.mark.parametrize("local", [False, True])+ @deferred_f_from_coro_f+ async def test_maxsize(+ self,+ server_url: str,+ dh: FTPDownloadHandler,+ tmp_path: Path,+ filename: str,+ local: bool,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ meta = {**self.req_meta, "download_maxsize": 10}+ if local:+ meta["ftp_local_filename"] = str(tmp_path / "local.txt").encode()+ request = Request(url=server_url + filename, meta=meta)+ with pytest.raises(DownloadCancelledError):+ await dh.download_request(request)+ assert "which is larger than download max size (10)" in caplog.text++ @deferred_f_from_coro_f+ async def test_maxsize_disabled(self, server_url: str, tmp_path: Path) -> None:+ crawler = get_crawler(settings_dict={"DOWNLOAD_MAXSIZE": 10})+ dh = build_from_crawler(FTPDownloadHandler, crawler)+ meta = {**self.req_meta, "download_maxsize": 0}+ request = Request(url=server_url + "file.txt", meta=meta)+ r = await dh.download_request(request)+ assert r.body == b"I have the power!"++ @pytest.mark.parametrize("filename", ["file.txt", "big.bin"])+ @deferred_f_from_coro_f+ async def test_warnsize(+ self,+ server_url: str,+ dh: FTPDownloadHandler,+ filename: str,+ caplog: pytest.LogCaptureFixture,+ ) -> None:+ meta = {**self.req_meta, "download_warnsize": 10}+ request = Request(url=server_url + filename, meta=meta)+ r = await dh.download_request(request)+ assert r.status == 200+ assert caplog.text.count("which is larger than download warn size (10)") == 1+@pytest.mark.parametrize(("filename", "response_class"),[diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.pyindex d96978a6a..bf5a64099 100644--- a/tests/test_downloadermiddleware_httpcompression.py+++ b/tests/test_downloadermiddleware_httpcompression.py@@ -814,6 +814,26 @@ class TestHttpCompression:),]+ @coroutine_test+ async def test_download_warnsize_setting_ignored_with_request_maxsize(+ self, caplog: pytest.LogCaptureFixture+ ) -> None:+ settings = {"DOWNLOAD_WARNSIZE": 10_000_000}+ crawler = get_crawler(Spider, settings_dict=settings)+ spider = crawler._create_spider("scrapytest.org")+ mw = build_from_crawler(HttpCompressionMiddleware, crawler)+ mw.open_spider(spider)+ response = self._getresponse("bomb-gzip")+ response.meta["download_maxsize"] = 20_000_000++ assert response.request+ caplog.clear()+ with caplog.at_level(+ WARNING, logger="scrapy.downloadermiddlewares.httpcompression"+ ):+ await mw.process_response(response.request, response)+ assert not caplog.records+@coroutine_testasync def test_download_warnsize_request_meta_br(self, caplog: pytest.LogCaptureFixturediff --git a/tests/test_spider_sitemap.py b/tests/test_spider_sitemap.pyindex 562e0337a..6f50e4547 100644--- a/tests/test_spider_sitemap.py+++ b/tests/test_spider_sitemap.py@@ -498,6 +498,23 @@ Sitemap: /sitemap-relative-url.xml),]+ def test_download_warnsize_setting_ignored_with_request_maxsize(+ self, caplog: pytest.LogCaptureFixture+ ) -> None:+ settings = {"DOWNLOAD_WARNSIZE": 10_000_000}+ crawler = get_crawler(settings_dict=settings)+ spider = self.spider_class.from_crawler(crawler, "example.com")+ body_path = Path(tests_datadir, "compressed", "bomb-gzip.bin")+ body = body_path.read_bytes()+ request = Request(+ url="https://example.com", meta={"download_maxsize": 20_000_000}+ )+ response = Response(url="https://example.com", body=body, request=request)+ caplog.clear()+ with caplog.at_level(WARNING, logger="scrapy.spiders.sitemap"):+ spider._get_sitemap_body(response)+ assert not caplog.records+def test_download_warnsize_request_meta(self, caplog: pytest.LogCaptureFixture) -> None:diff --git a/tests/utils/bases/download_handlers_http.py b/tests/utils/bases/download_handlers_http.pyindex 6a3abd0ae..e1158cac8 100644--- a/tests/utils/bases/download_handlers_http.py+++ b/tests/utils/bases/download_handlers_http.py@@ -787,6 +787,31 @@ class TestHttpBase(ABC):)assert caplog.text.count("download warn size (4)") == 1+ @pytest.mark.parametrize(+ ("meta", "warns"),+ [+ pytest.param({"download_maxsize": 100}, False, id="maxsize-only"),+ pytest.param(+ {"download_maxsize": 100, "download_warnsize": 4},+ True,+ id="maxsize-and-warnsize",+ ),+ ],+ )+ @coroutine_test+ async def test_download_warnsize_with_request_maxsize(+ self,+ caplog: pytest.LogCaptureFixture,+ mockserver: MockServer,+ meta: dict[str, int],+ warns: bool,+ ) -> None:+ request = Request(mockserver.url("/text", is_secure=self.is_secure), meta=meta)+ async with self.get_dh({"DOWNLOAD_WARNSIZE": 4}) as download_handler:+ response = await download_handler.download_request(request)+ assert response.body == b"Works"+ assert ("download warn size (4)" in caplog.text) is warns+@coroutine_testasync def test_download_with_warnsize_no_content_length(self, caplog: pytest.LogCaptureFixture, mockserver: MockServerdiff --git a/tox.ini b/tox.iniindex cc7370b19..56326cb2e 100644--- a/tox.ini+++ b/tox.ini@@ -74,6 +74,7 @@ deps =Pillow==12.3.0Protego==0.7.0Twisted==26.4.0+ aioftp==0.28.3aiohttp==3.14.3attrs==26.1.0backports.zstd==1.7.0@@ -88,6 +89,7 @@ deps =ipython==9.17.1pyOpenSSL==26.4.0pytest==9.1.1+ python-socks[asyncio]==3.1.1socksio==1.0.0types-Pygments==2.21.0.20260819types-defusedxml==0.7.0.20260504@@ -140,6 +142,7 @@ deps =pytest==8.4.0Protego==0.2.0Twisted==21.7.0+ aioftp==0.26.1aiohttp==3.13.3backports.zstd==1.3.0brotli==1.2.0; implementation_name != "pypy"@@ -183,6 +186,8 @@ deps ={[testenv]deps}PillowTwisted[http2]+ python-socks[asyncio]+ siosocks # SOCKS server for testsboto3bpython # optional for shell wrapper testscolorlog@@ -200,6 +205,8 @@ deps ={[min]deps}Pillow==9.2.0Twisted[http2]==21.7.0+ python-socks[asyncio]==2.6.0+ siosocks==0.2.0 # SOCKS server for testsboto3==1.35.2bpython==0.7.1colorlog==6.6.0@@ -236,6 +243,7 @@ commands_pre =git+https://github.com/python-pillow/Pillow \git+https://github.com/MagicStack/uvloop \git+https://github.com/aio-libs/aiohttp \+ git+https://github.com/aio-libs/aioftp \git+https://github.com/pyca/cryptography \git+https://github.com/scrapy/cssselect \git+https://github.com/tiran/defusedxml \@@ -324,6 +332,7 @@ basepython = pypy3.11deps =PyPyDispatcher==2.1.0{[test-requirements]deps}+ aioftp==0.26.1aiohttp==3.13.3pytest==8.4.0Protego==0.2.0