Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add an FTP download handler for reactorless mode (#8213)

Adrian53b34f729334
diff --git a/docs/intro/install.rst b/docs/intro/install.rst
index 2efdf9f10..faeb4f927 100644
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@@ -115,6 +115,8 @@ The following extras are available:
- Reading the system clipboard with :command:`genrequest`
* - ``color``
- :setting:`LOG_COLOR`
+ * - ``ftp-socks``
+ - SOCKS proxy support in :ref:`aioftp-handler`
* - ``gcs``
- :ref:`Google Cloud Storage <topics-feed-storage-gcs>` for
:ref:`feed exports <topics-feed-exports>` and
diff --git a/docs/news.rst b/docs/news.rst
index c133a5f53..fd007243b 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -69,6 +69,16 @@ Backward-incompatible changes
effect.
(:gh:`8181`)
+- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now
+ enforces :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE`, also
+ when using :reqmeta:`ftp_local_filename`, so downloads bigger than 1 GiB
+ now fail by default. To download bigger files, raise the limit to a size
+ that fits them, with :setting:`DOWNLOAD_MAXSIZE` or with the
+ :reqmeta:`download_maxsize` request meta key for specific requests. You can
+ also set the limit to ``0`` to lift it altogether.
+
+ (:gh:`8213`)
+
.. _release-2.19.0:
Scrapy 2.19.0 (2026-09-10)
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index b3cd35f10..643e274ec 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -189,7 +189,8 @@ in future Scrapy versions. The following features are not available:
is likely the biggest difference; Scrapy provides an HTTP(S) download handler
that doesn't require a reactor and will be used instead of it:
:class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`)
-* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`
+* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` (replaced
+ by :class:`~scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler`)
* :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`
* :ref:`topics-telnetconsole`
* :class:`~scrapy.crawler.CrawlerRunner` and
@@ -215,7 +216,8 @@ the defaults of some other settings:
* :setting:`TELNETCONSOLE_ENABLED` is set to ``False``.
* The ``"http"`` and ``"https"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` are
set to ``"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"``.
-* The ``"ftp"`` key in :setting:`DOWNLOAD_HANDLERS_BASE` is set to ``None``.
+* The ``"ftp"`` and ``"ftps"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` are
+ set to ``"scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler"``.
Thus, :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
is used by default for making HTTP(S) requests. Please refer to its
diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst
index 19e5de2e3..ab87a0ebb 100644
--- a/docs/topics/download-handlers.rst
+++ b/docs/topics/download-handlers.rst
@@ -450,6 +450,51 @@ Whether to enable HTTP/2 support in this handler.
Built-in non-HTTP download handlers reference
=============================================
+.. _aioftp-handler:
+
+AioftpDownloadHandler
+---------------------
+
+.. versionadded:: VERSION
+
+.. autoclass:: scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler
+
+| Supported schemes: ``ftp``, ``ftps``.
+| :ref:`Lazy <lazy-download-handlers>`: yes.
+| :ref:`Requires asyncio support <using-asyncio>`: yes.
+| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.
+
+This handler supports ``ftp://host/path`` FTP URIs and ``ftps://host/path``
+FTPS URIs. FTPS uses `implicit TLS`_ on port 990 and `explicit TLS`_ on any
+other port.
+
+.. _implicit TLS: https://en.wikipedia.org/wiki/FTPS#Implicit
+.. _explicit TLS: https://en.wikipedia.org/wiki/FTPS#Explicit
+
+It's implemented using the aioftp_ library.
+
+.. _aioftp: https://aioftp.rtfd.io/
+
+A path ending in ``/`` is listed instead of downloaded. The response body is
+the plain-text output of the FTP ``NLST`` command, one file or directory name
+per line.
+
+It supports SOCKS4 and SOCKS5 :reqmeta:`proxies <proxy>` if you install the
+:ref:`ftp-socks <extras>` extra.
+
+.. warning::
+
+ This handler is experimental, and not yet recommended for production
+ environments. Future Scrapy versions may introduce related changes without
+ a deprecation period or warning or even remove it altogether.
+
+Limitations:
+
+- Only passive mode is supported: the handler cannot be used if
+ :setting:`FTP_PASSIVE_MODE` is ``False``, and requests that set the
+ :reqmeta:`ftp_passive` request meta key to ``False`` are downloaded in
+ passive mode, with an error logged.
+
DataURIDownloadHandler
----------------------
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index b96ac5f4e..204e0d338 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1122,7 +1122,8 @@ Default:
"http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
- "ftp": None,
+ "ftp": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",
+ "ftps": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",
}
(when :setting:`TWISTED_REACTOR_ENABLED` is ``False``)
@@ -1256,7 +1257,8 @@ Use ``0`` to disable this limit.
.. note::
This limit can be set per-request using the :reqmeta:`download_warnsize`
- :attr:`.Request.meta` key.
+ :attr:`.Request.meta` key. Requests that set :reqmeta:`download_maxsize`
+ but not :reqmeta:`download_warnsize` log no warning.
.. note::
diff --git a/pyproject.toml b/pyproject.toml
index ad402b3ec..602ee6041 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,6 +8,7 @@ dynamic = ["version"]
description = "A high-level Web Crawling and Web Scraping framework"
dependencies = [
"Twisted>=21.7.0",
+ "aioftp>=0.26.1",
"aiohttp>=3.13.3",
"charset-normalizer>=3.4.0",
"cryptography>=41.0.5",
@@ -72,6 +73,7 @@ Tracker = "https://github.com/scrapy/scrapy/issues"
bpython = ["bpython>=0.7.1"]
clipboard = ["pyperclip>=1.8.0"]
color = ["colorlog>=6.6.0"]
+ftp-socks = ["python-socks[asyncio]>=2.6.0"]
gcs = ["google-cloud-storage>=1.30.0"]
httpx = ["httpx2[http2,socks]>=2.0.0"]
images = ["Pillow>=9.2.0"]
@@ -153,6 +155,7 @@ module = [
"pyperclip",
"pytest_twisted",
"robotexclusionrulesparser",
+ "siosocks.*",
"zope.interface.*",
]
ignore_missing_imports = true
diff --git a/scrapy/core/_http2/stream.py b/scrapy/core/_http2/stream.py
index 8dca37e2e..ab99d68c8 100644
--- a/scrapy/core/_http2/stream.py
+++ b/scrapy/core/_http2/stream.py
@@ -19,6 +19,7 @@ from scrapy.http.headers import Headers
from scrapy.utils._download_handlers import (
check_stop_download,
get_maxsize_msg,
+ get_warnsize,
get_warnsize_msg,
make_response,
)
@@ -126,9 +127,7 @@ class Stream:
self._download_maxsize = self._request.meta.get(
"download_maxsize", download_maxsize
)
- self._download_warnsize = self._request.meta.get(
- "download_warnsize", download_warnsize
- )
+ self._download_warnsize = get_warnsize(self._request.meta, download_warnsize)
# Metadata of an HTTP/2 connection stream
# initialized when stream is instantiated
@@ -190,7 +189,7 @@ class Stream:
self._response["headers"].get(b"Content-Length", -1)
)
return (
- self._download_warnsize
+ bool(self._download_warnsize)
and (
self._response["flow_controlled_size"] > self._download_warnsize
or content_length_header > self._download_warnsize
diff --git a/scrapy/core/downloader/handlers/_aioftp.py b/scrapy/core/downloader/handlers/_aioftp.py
new file mode 100644
index 000000000..e365123fc
--- /dev/null
+++ b/scrapy/core/downloader/handlers/_aioftp.py
@@ -0,0 +1,274 @@
+from __future__ import annotations
+
+import asyncio
+import ipaddress
+import logging
+import socket
+import ssl
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, BinaryIO, ClassVar, NamedTuple
+from urllib.parse import unquote, urlsplit
+
+import aioftp
+from aioftp.common import SSLSessionBoundContext
+
+from scrapy.exceptions import (
+ CannotResolveHostError,
+ DownloadConnectionRefusedError,
+ DownloadFailedError,
+ DownloadTimeoutError,
+ NotConfigured,
+)
+from scrapy.http import Headers
+from scrapy.utils._ssl import _log_sslobj_debug_info, _make_ssl_context
+from scrapy.utils.httpobj import urlparse_cached
+
+from ._base_streaming import BaseStreamingDownloadHandler, _BaseResponseArgs
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncGenerator, AsyncIterator
+
+ from scrapy.crawler import Crawler
+ from scrapy.http import Response
+ from scrapy.http.request import Request
+
+
+logger = logging.getLogger(__name__)
+
+_TIMEOUT_EXCEPTIONS: tuple[type[Exception], ...] = (TimeoutError,)
+_CONNECTION_REFUSED_EXCEPTIONS: tuple[type[Exception], ...] = (ConnectionRefusedError,)
+_DOWNLOAD_FAILED_EXCEPTIONS: tuple[type[Exception], ...] = (
+ aioftp.AIOFTPException,
+ ConnectionResetError,
+)
+_HAS_SOCKS = False
+try:
+ from python_socks import (
+ ProxyConnectionError,
+ ProxyError,
+ ProxyTimeoutError,
+ ProxyType,
+ )
+ from python_socks.async_.asyncio import Proxy
+
+ _HAS_SOCKS = True
+ _TIMEOUT_EXCEPTIONS += (ProxyTimeoutError,)
+ _CONNECTION_REFUSED_EXCEPTIONS += (ProxyConnectionError,)
+ _DOWNLOAD_FAILED_EXCEPTIONS += (ProxyError,)
+except ImportError: # pragma: no cover
+ pass
+
+
+class _ProxyClient(aioftp.Client):
+ """An :class:`aioftp.Client` that opens its connections through
+ *proxy*."""
+
+ def __init__(self, proxy: Proxy, **kwargs: Any):
+ super().__init__(**kwargs)
+ self._proxy = proxy
+
+ async def _open_connection(
+ self, host: str, port: int
+ ) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]:
+ ssl_context = self.ssl
+ if self._stream is not None and (
+ ssl_object := self._stream.writer.get_extra_info("ssl_object")
+ ):
+ # data connections reuse the TLS session of the control connection
+ ssl_context = SSLSessionBoundContext(
+ ssl.PROTOCOL_TLS_CLIENT,
+ context=ssl_object.context,
+ session=ssl_object.session,
+ )
+ sock = await self._proxy.connect(host, port, timeout=self.connection_timeout)
+ return await asyncio.open_connection(
+ sock=sock,
+ ssl=ssl_context or None,
+ server_hostname=host if ssl_context else None,
+ )
+
+
+class _AioftpResponse(NamedTuple):
+ status: int # mapped to common HTTP status codes
+ control: aioftp.ThrottleStreamIO
+ data: aioftp.DataConnectionThrottleStreamIO | None
+
+
+class AioftpDownloadHandler(BaseStreamingDownloadHandler[_AioftpResponse]):
+ lazy = True
+ experimental: ClassVar[bool] = True
+
+ def __init__(self, crawler: Crawler):
+ super().__init__(crawler)
+ # aioftp only supports passive mode
+ if not crawler.settings.getbool("FTP_PASSIVE_MODE"):
+ raise NotConfigured(
+ f"{type(self).__name__} does not support disabling FTP_PASSIVE_MODE."
+ )
+ self._user: str = crawler.settings["FTP_USER"]
+ self._password: str = crawler.settings["FTP_PASSWORD"]
+ self._ssl_context: ssl.SSLContext = _make_ssl_context(crawler.settings)
+
+ @asynccontextmanager
+ async def _make_request(
+ self,
+ request: Request,
+ timeout: float, # noqa: ASYNC109
+ ) -> AsyncGenerator[_AioftpResponse]:
+ url = urlparse_cached(request)
+ assert url.hostname
+ if not request.meta.get("ftp_passive", True):
+ logger.error(
+ f"{type(self).__name__} does not support active mode, so the"
+ f" 'ftp_passive' request meta key will be ignored for {request}."
+ )
+
+ implicit_tls = url.scheme == "ftps" and url.port == 990
+ client_kwargs: dict[str, Any] = {
+ "socket_timeout": timeout,
+ "connection_timeout": timeout,
+ "ssl": self._ssl_context if implicit_tls else None,
+ }
+ proxy = self._get_proxy(request)
+ client = (
+ _ProxyClient(proxy, **client_kwargs)
+ if proxy
+ else aioftp.Client(**client_kwargs)
+ )
+ try:
+ await client.connect(url.hostname, url.port or 21)
+ if url.scheme == "ftps" and not implicit_tls:
+ await client.upgrade_to_tls(self._ssl_context)
+ await client.login(
+ request.meta.get("ftp_user", self._user),
+ request.meta.get("ftp_password", self._password),
+ )
+ reader, writer = await client.get_passive_connection()
+ data = aioftp.DataConnectionThrottleStreamIO(
+ client, reader, writer, throttles={}, timeout=timeout
+ )
+ try:
+ path = unquote(url.path)
+ command = "NLST" if not path or path.endswith("/") else "RETR"
+ await client.command(f"{command} {path}".rstrip(), "1xx")
+ except aioftp.StatusCodeError as e:
+ data.close()
+ status = 404 if e.received_codes[-1].matches("550") else 503
+ yield _AioftpResponse(status, client.stream, None)
+ else:
+ try:
+ yield _AioftpResponse(200, client.stream, data)
+ finally:
+ data.close()
+ except _TIMEOUT_EXCEPTIONS as e:
+ raise DownloadTimeoutError(
+ f"Getting {request.url} took longer than {timeout} seconds."
+ ) from e
+ except _CONNECTION_REFUSED_EXCEPTIONS as e:
+ raise DownloadConnectionRefusedError(str(e)) from e
+ except socket.gaierror as e:
+ raise CannotResolveHostError(str(e)) from e
+ except _DOWNLOAD_FAILED_EXCEPTIONS as e:
+ raise DownloadFailedError(str(e)) from e
+ finally:
+ client.close()
+
+ def _get_proxy(self, request: Request) -> Proxy | None:
+ proxy_url = self._extract_proxy_url_with_creds(request)
+ if not proxy_url:
+ return None
+ parts = urlsplit(proxy_url)
+ if parts.scheme not in ("socks4", "socks5"):
+ raise NotImplementedError(
+ f"{type(self).__name__} only supports SOCKS proxies, got {proxy_url!r}."
+ )
+ if not _HAS_SOCKS: # pragma: no cover
+ raise ValueError(
+ f"SOCKS proxy support in {type(self).__name__} requires the"
+ f" 'ftp-socks' extra to be installed."
+ )
+ assert parts.hostname
+ return Proxy(
+ ProxyType.SOCKS4 if parts.scheme == "socks4" else ProxyType.SOCKS5,
+ parts.hostname,
+ parts.port or 1080,
+ unquote(parts.username) if parts.username else None,
+ unquote(parts.password) if parts.password else None,
+ )
+
+ @staticmethod
+ def _local_filename(response: _AioftpResponse, request: Request) -> bytes | None:
+ if response.data is None:
+ return None
+ return request.meta.get("ftp_local_filename")
+
+ def _open_body_buffer(
+ self, response: _AioftpResponse, request: Request
+ ) -> BinaryIO:
+ if local_filename := self._local_filename(response, request):
+ return Path(local_filename.decode()).open("wb")
+ return super()._open_body_buffer(response, request)
+
+ def _get_body(
+ self, buffer: BinaryIO, response: _AioftpResponse, request: Request
+ ) -> bytes:
+ if local_filename := self._local_filename(response, request):
+ return local_filename
+ return super()._get_body(buffer, response, request)
+
+ async def _read_response(
+ self, response: _AioftpResponse, request: Request
+ ) -> Response:
+ result = await super()._read_response(response, request)
+ if response.data is not None:
+ self._add_ftp_headers(result, response, request)
+ return result
+
+ def _add_ftp_headers(
+ self, result: Response, response: _AioftpResponse, request: Request
+ ) -> None:
+ if local_filename := self._local_filename(response, request):
+ size = Path(local_filename.decode()).stat().st_size
+ else:
+ size = len(result.body)
+ result.headers.update({"Local Filename": local_filename or b"", "Size": size})
+
+ @staticmethod
+ def _extract_headers(response: _AioftpResponse) -> Headers:
+ return Headers()
+
+ @staticmethod
+ def _build_base_response_args(
+ response: _AioftpResponse,
+ request: Request,
+ headers: Headers,
+ ) -> _BaseResponseArgs:
+ peername = response.control.writer.get_extra_info("peername")
+ ssl_object = response.control.writer.get_extra_info("ssl_object")
+ cert = ssl_object.getpeercert(binary_form=True) if ssl_object else None
+ return {
+ "status": response.status,
+ "url": request.url,
+ "certificate": cert,
+ "protocol": "FTP",
+ "ip_address": ipaddress.ip_address(peername[0]),
+ "headers": headers,
+ }
+
+ @staticmethod
+ async def _iter_body_chunks(response: _AioftpResponse) -> AsyncIterator[bytes]:
+ if response.data is None:
+ return
+ async for chunk in response.data.iter_by_block():
+ yield chunk
+ # raises StatusCodeError if the server reports a failed transfer
+ await response.data.finish()
+
+ @staticmethod
+ def _is_dataloss_exception(exc: Exception) -> bool:
+ return isinstance(exc, aioftp.StatusCodeError)
+
+ def _log_tls_info(self, response: _AioftpResponse, request: Request) -> None:
+ if ssl_object := response.control.writer.get_extra_info("ssl_object"):
+ _log_sslobj_debug_info(ssl_object)
diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.py
index 596942169..1888e7a26 100644
--- a/scrapy/core/downloader/handlers/_base_streaming.py
+++ b/scrapy/core/downloader/handlers/_base_streaming.py
@@ -8,6 +8,7 @@ from io import BytesIO
from typing import (
TYPE_CHECKING,
Any,
+ BinaryIO,
ClassVar,
Generic,
NoReturn,
@@ -27,6 +28,7 @@ from scrapy.utils._download_handlers import (
check_stop_download,
get_dataloss_msg,
get_maxsize_msg,
+ get_warnsize,
get_warnsize_msg,
make_response,
normalize_bind_address,
@@ -140,6 +142,17 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon
def _log_tls_info(self, response: _ResponseT, request: Request) -> None:
"""Log TLS connection details, if possible."""
+ def _open_body_buffer(self, response: _ResponseT, request: Request) -> BinaryIO:
+ """Return a file object to write the response body into."""
+ return BytesIO()
+
+ def _get_body(
+ self, buffer: BinaryIO, response: _ResponseT, request: Request
+ ) -> bytes:
+ """Return the response body given the *buffer* it was written into."""
+ assert isinstance(buffer, BytesIO)
+ return buffer.getvalue()
+
async def download_request(self, request: Request) -> Response:
if not self.supports_proxies and request.meta.get("proxy"):
raise NotImplementedError(f"{type(self).__name__} doesn't support proxies.")
@@ -160,7 +173,7 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon
async def _read_response(self, response: _ResponseT, request: Request) -> Response:
maxsize: int = request.meta.get("download_maxsize", self._default_maxsize)
- warnsize: int = request.meta.get("download_warnsize", self._default_warnsize)
+ warnsize = get_warnsize(request.meta, self._default_warnsize)
headers = self._extract_headers(response)
content_length = headers.get("Content-Length")
@@ -194,56 +207,56 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon
stop_download=stop_download,
)
- response_body = BytesIO()
- bytes_received = 0
- try:
- async for chunk in self._iter_body_chunks(response):
- response_body.write(chunk)
- bytes_received += len(chunk)
+ with self._open_body_buffer(response, request) as response_body:
+ bytes_received = 0
+ try:
+ async for chunk in self._iter_body_chunks(response):
+ response_body.write(chunk)
+ bytes_received += len(chunk)
+
+ if stop_download := check_stop_download(
+ signals.bytes_received, self.crawler, request, data=chunk
+ ):
+ return make_response(
+ **make_response_base_args,
+ body=self._get_body(response_body, response, request),
+ stop_download=stop_download,
+ )
+
+ if maxsize and bytes_received > maxsize:
+ response_body.truncate(0)
+ self._cancel_maxsize(
+ bytes_received, maxsize, request, expected=False
+ )
- if stop_download := check_stop_download(
- signals.bytes_received, self.crawler, request, data=chunk
- ):
+ if warnsize and bytes_received > warnsize and not reached_warnsize:
+ reached_warnsize = True
+ logger.warning(
+ get_warnsize_msg(
+ bytes_received, warnsize, request, expected=False
+ )
+ )
+ except Exception as e:
+ if not self._is_dataloss_exception(e):
+ raise
+ fail_on_dataloss: bool = request.meta.get(
+ "download_fail_on_dataloss", self._fail_on_dataloss
+ )
+ if not fail_on_dataloss:
return make_response(
**make_response_base_args,
- body=response_body.getvalue(),
- stop_download=stop_download,
- )
-
- if maxsize and bytes_received > maxsize:
- response_body.truncate(0)
- self._cancel_maxsize(
- bytes_received, maxsize, request, expected=False
+ body=self._get_body(response_body, response, request),
+ flags=["dataloss"],
)
+ if not self._fail_on_dataloss_warned:
+ logger.warning(get_dataloss_msg(request.url))
+ self._fail_on_dataloss_warned = True
+ raise ResponseDataLossError(str(e)) from e
- if warnsize and bytes_received > warnsize and not reached_warnsize:
- reached_warnsize = True
- logger.warning(
- get_warnsize_msg(
- bytes_received, warnsize, request, expected=False
- )
- )
- except Exception as e:
- if not self._is_dataloss_exception(e):
- raise
- fail_on_dataloss: bool = request.meta.get(
- "download_fail_on_dataloss", self._fail_on_dataloss
+ return make_response(
+ **make_response_base_args,
+ body=self._get_body(response_body, response, request),
)
- if not fail_on_dataloss:
- return make_response(
- **make_response_base_args,
- body=response_body.getvalue(),
- flags=["dataloss"],
- )
- if not self._fail_on_dataloss_warned:
- logger.warning(get_dataloss_msg(request.url))
- self._fail_on_dataloss_warned = True
- raise ResponseDataLossError(str(e)) from e
-
- return make_response(
- **make_response_base_args,
- body=response_body.getvalue(),
- )
@staticmethod
def _request_headers(request: Request) -> Headers:
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
index 327193a29..e3b8f6331 100644
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -30,6 +30,7 @@ In case of status 200 request, response.headers will come with two keys:
from __future__ import annotations
+import logging
import re
from io import BytesIO
from pathlib import Path
@@ -39,9 +40,14 @@ from urllib.parse import unquote
from twisted.internet.protocol import ClientCreator, Protocol
from scrapy.core.downloader.handlers.base import BaseDownloadHandler
-from scrapy.exceptions import DownloadFailedError, NotConfigured
+from scrapy.exceptions import DownloadCancelledError, DownloadFailedError, NotConfigured
from scrapy.http import Response
from scrapy.responsetypes import responsetypes
+from scrapy.utils._download_handlers import (
+ get_maxsize_msg,
+ get_warnsize,
+ get_warnsize_msg,
+)
from scrapy.utils.defer import maybe_deferred_to_future
from scrapy.utils.httpobj import urlparse_cached
@@ -52,17 +58,47 @@ if TYPE_CHECKING:
from scrapy.crawler import Crawler
+logger = logging.getLogger(__name__)
+
+
class ReceivedDataProtocol(Protocol):
- def __init__(self, filename: bytes | None = None):
+ def __init__(
+ self,
+ filename: bytes | None = None,
+ *,
+ request: Request | None = None,
+ maxsize: int = 0,
+ warnsize: int = 0,
+ ):
self.__filename: bytes | None = filename
self.body: BinaryIO = (
Path(filename.decode()).open("wb") if filename else BytesIO()
)
self.size: int = 0
+ self._request: Request | None = request
+ self._maxsize: int = maxsize
+ self._warnsize: int = warnsize
+ self._exceeded_maxsize: bool = False
+ self._reached_warnsize: bool = False
def dataReceived(self, data: bytes) -> None:
self.body.write(data)
self.size += len(data)
+ if self._maxsize and self.size > self._maxsize:
+ self._exceeded_maxsize = True
+ self.body.truncate(0)
+ assert self.transport
+ self.transport.loseConnection()
+ elif (
+ self._warnsize and self.size > self._warnsize and not self._reached_warnsize
+ ):
+ self._reached_warnsize = True
+ assert self._request is not None
+ logger.warning(
+ get_warnsize_msg(
+ self.size, self._warnsize, self._request, expected=False
+ )
+ )
@property
def filename(self) -> bytes | None:
@@ -91,6 +127,8 @@ class FTPDownloadHandler(BaseDownloadHandler):
self.default_user = crawler.settings["FTP_USER"]
self.default_password = crawler.settings["FTP_PASSWORD"]
self.passive_mode = crawler.settings["FTP_PASSIVE_MODE"]
+ self._default_maxsize: int = crawler.settings.getint("DOWNLOAD_MAXSIZE")
+ self._default_warnsize: int = crawler.settings.getint("DOWNLOAD_WARNSIZE")
async def download_request(self, request: Request) -> Response:
from twisted.internet import reactor
@@ -110,26 +148,41 @@ class FTPDownloadHandler(BaseDownloadHandler):
)
filepath = unquote(parsed_url.path)
is_listing = filepath == "" or filepath.endswith("/")
- protocol = ReceivedDataProtocol(request.meta.get("ftp_local_filename"))
+ maxsize: int = request.meta.get("download_maxsize", self._default_maxsize)
+ protocol = ReceivedDataProtocol(
+ request.meta.get("ftp_local_filename"),
+ request=request,
+ maxsize=maxsize,
+ warnsize=get_warnsize(request.meta, self._default_warnsize),
+ )
try:
if is_listing:
await maybe_deferred_to_future(client.nlst(filepath, protocol))
else:
await maybe_deferred_to_future(client.retrieveFile(filepath, protocol))
except CommandFailed as e:
- message = str(e)
- # Twisted only raises CommandFailed for a reply whose numeric code
- # it has parsed, so the message always carries that code.
- m = _CODE_RE.search(message)
- assert m
- httpcode = self.CODE_MAPPING.get(m.group(), self.CODE_MAPPING["default"])
- return Response(url=request.url, status=httpcode, body=message.encode())
+ if not protocol._exceeded_maxsize:
+ message = str(e)
+ # Twisted only raises CommandFailed for a reply whose numeric
+ # code it has parsed, so the message always carries that code.
+ m = _CODE_RE.search(message)
+ assert m
+ httpcode = self.CODE_MAPPING.get(
+ m.group(), self.CODE_MAPPING["default"]
+ )
+ return Response(url=request.url, status=httpcode, body=message.encode())
except ConnectionLost as e:
raise DownloadFailedError(str(e)) from e
finally:
protocol.close()
assert client.transport
client.transport.loseConnection()
+ if protocol._exceeded_maxsize:
+ warning_msg = get_maxsize_msg(
+ protocol.size, maxsize, request, expected=False
+ )
+ logger.warning(warning_msg)
+ raise DownloadCancelledError(warning_msg)
headers = {"Local Filename": protocol.filename or b"", "Size": protocol.size}
body = protocol.filename or protocol.body.read()
respcls = responsetypes.from_args(url=request.url, body=body)
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
index 7146143f1..bcb32a076 100644
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -52,6 +52,7 @@ from scrapy.utils._download_handlers import (
check_stop_download,
get_dataloss_msg,
get_maxsize_msg,
+ get_warnsize,
get_warnsize_msg,
make_response,
normalize_bind_address,
@@ -585,7 +586,7 @@ class _ScrapyAgent:
}
maxsize = request.meta.get("download_maxsize", self._maxsize)
- warnsize = request.meta.get("download_warnsize", self._warnsize)
+ warnsize = get_warnsize(request.meta, self._warnsize)
expected_size = (
cast("int", txresponse.length)
if txresponse.length != UNKNOWN_LENGTH
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index fd75fba9a..cb42126a8 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -283,7 +283,10 @@ class Crawler:
self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = (
"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"
)
- self.settings["DOWNLOAD_HANDLERS_BASE"]["ftp"] = None
+ for scheme in ("ftp", "ftps"):
+ self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = (
+ "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler"
+ )
# Cannot use @deferred_f_from_coro_f because that relies on the reactor
# being installed already, which is done within _apply_settings(), inside
diff --git a/scrapy/downloadermiddlewares/httpcompression.py b/scrapy/downloadermiddlewares/httpcompression.py
index 5ab9eac12..8beee247e 100644
--- a/scrapy/downloadermiddlewares/httpcompression.py
+++ b/scrapy/downloadermiddlewares/httpcompression.py
@@ -22,6 +22,7 @@ from scrapy.utils._compression import (
_unzstd,
gunzip,
)
+from scrapy.utils._download_handlers import get_warnsize
from scrapy.utils.asyncio import run_in_thread
from scrapy.utils.decorators import _warn_spider_arg
from scrapy.utils.deprecate import warn_on_deprecated_spider_attribute
@@ -101,7 +102,7 @@ class HttpCompressionMiddleware:
content_encoding = response.headers.getlist("Content-Encoding")
if content_encoding:
max_size = request.meta.get("download_maxsize", self._max_size)
- warn_size = request.meta.get("download_warnsize", self._warn_size)
+ warn_size = get_warnsize(request.meta, self._warn_size)
try:
decoded_body, content_encoding = await self._handle_encoding(
response.body, content_encoding, max_size
diff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.py
index 425466a27..0c20c1e1d 100644
--- a/scrapy/spiders/sitemap.py
+++ b/scrapy/spiders/sitemap.py
@@ -246,7 +246,10 @@ class SitemapSpider(Spider):
if gzip_magic_number(response):
uncompressed_size = len(response.body)
max_size = response.meta.get("download_maxsize", self._max_size)
- warn_size = response.meta.get("download_warnsize", self._warn_size)
+ # circular import
+ from scrapy.utils._download_handlers import get_warnsize # noqa: PLC0415
+
+ warn_size = get_warnsize(response.meta, self._warn_size)
try:
body = gunzip(response.body, max_size=max_size)
except _DecompressionMaxSizeExceeded:
diff --git a/scrapy/utils/_download_handlers.py b/scrapy/utils/_download_handlers.py
index 23b715ac6..57330bd2b 100644
--- a/scrapy/utils/_download_handlers.py
+++ b/scrapy/utils/_download_handlers.py
@@ -129,6 +129,20 @@ def get_maxsize_msg(size: int, limit: int, request: Request, *, expected: bool)
)
+def get_warnsize(meta: dict[str, Any], default: int) -> int:
+ """Return the download warn size for a request with the given *meta*.
+
+ A request-specific ``download_maxsize`` disables the *default* warn size,
+ since a user who sets a request-specific size limit expects that size.
+ """
+ if "download_warnsize" in meta:
+ warnsize: int = meta["download_warnsize"]
+ return warnsize
+ if "download_maxsize" in meta:
+ return 0
+ return default
+
+
def get_warnsize_msg(size: int, limit: int, request: Request, *, expected: bool) -> str:
prefix = "Expected to receive" if expected else "Received"
return (
diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py
index d461056e5..8f11e70aa 100644
--- a/scrapy/utils/test.py
+++ b/scrapy/utils/test.py
@@ -50,7 +50,8 @@ def get_reactor_settings() -> dict[str, Any]:
# last one from them.
settings["TWISTED_REACTOR_ENABLED"] = False
settings["DOWNLOAD_HANDLERS"] = {
- "ftp": None,
+ "ftp": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",
+ "ftps": "scrapy.core.downloader.handlers._aioftp.AioftpDownloadHandler",
"http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
}
diff --git a/tests/mockserver/ftp.py b/tests/mockserver/ftp.py
index d09bf6115..2398466c6 100644
--- a/tests/mockserver/ftp.py
+++ b/tests/mockserver/ftp.py
@@ -7,6 +7,7 @@ from pathlib import Path
from shutil import rmtree
from subprocess import PIPE, Popen
from tempfile import mkdtemp
+from threading import Thread
from typing import TYPE_CHECKING, Self
from pyftpdlib.authorizers import DummyAuthorizer
@@ -29,6 +30,7 @@ class MockFTPServer:
"""
proc: Popen[str]
+ _stderr_reader: Thread
port: int
path: Path
@@ -56,6 +58,9 @@ class MockFTPServer:
raise RuntimeError(
"The FTP server failed to start or the output is unrecognized"
)
+ # Keep the pipe from filling up, which would block the server.
+ self._stderr_reader = Thread(target=self.proc.stderr.read, daemon=True)
+ self._stderr_reader.start()
return self
def __exit__(
@@ -66,6 +71,7 @@ class MockFTPServer:
) -> None:
rmtree(str(self.path))
self.proc.kill()
+ self._stderr_reader.join()
self.proc.communicate()
def url(self, path: str) -> str:
diff --git a/tests/test_downloader_handler_aioftp.py b/tests/test_downloader_handler_aioftp.py
new file mode 100644
index 000000000..bf98920f7
--- /dev/null
+++ b/tests/test_downloader_handler_aioftp.py
@@ -0,0 +1,409 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+from contextlib import asynccontextmanager
+from ipaddress import ip_address
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from w3lib.http import basic_auth_header
+
+from scrapy.core.downloader.handlers._aioftp import _HAS_SOCKS, AioftpDownloadHandler
+from scrapy.exceptions import (
+ CannotResolveHostError,
+ DownloadCancelledError,
+ DownloadConnectionRefusedError,
+ DownloadFailedError,
+ DownloadTimeoutError,
+ NotConfigured,
+ ResponseDataLossError,
+)
+from scrapy.http import HtmlResponse, Request, TextResponse
+from scrapy.utils.misc import build_from_crawler
+from scrapy.utils.test import get_crawler
+from tests.mockserver.ftp import MockFTPServer
+from tests.utils.decorators import coroutine_test
+
+try:
+ from siosocks.io.asyncio import socks_server_handler
+except ImportError:
+ socks_server_handler = None
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncGenerator, Callable, Generator
+ from pathlib import Path
+
+ from scrapy.http import Response
+
+pytestmark = pytest.mark.only_asyncio
+
+FILES = {
+ "file.txt": b"I have the power!",
+ "file with spaces.txt": b"Moooooooooo power!",
+ "html-file-without-extension": b"<!DOCTYPE html>\n<title>.</title>",
+}
+
+
+def _write_files(server: MockFTPServer) -> MockFTPServer:
+ for name, content in FILES.items():
+ (server.path / name).write_bytes(content)
+ return server
+
+
[email protected](scope="module")
+def ftp_server() -> Generator[MockFTPServer]:
+ with MockFTPServer() as server:
+ yield _write_files(server)
+
+
[email protected](scope="module")
+def ftps_server() -> Generator[MockFTPServer]:
+ with MockFTPServer(tls=True) as server:
+ yield _write_files(server)
+
+
+async def _download(
+ request: Request, settings: dict[str, Any] | None = None
+) -> Response:
+ crawler = get_crawler(settings_dict=settings)
+ dh = build_from_crawler(AioftpDownloadHandler, crawler)
+ try:
+ return await dh.download_request(request)
+ finally:
+ await dh.close()
+
+
+ ("path", "response_class"),
+ [
+ ("file.txt", TextResponse),
+ ("file with spaces.txt", TextResponse),
+ ("html-file-without-extension", HtmlResponse),
+ ],
+)
+@coroutine_test
+async def test_download(
+ ftp_server: MockFTPServer, path: str, response_class: type[Response]
+) -> None:
+ response = await _download(Request(ftp_server.url(path)))
+ assert type(response) is response_class
+ assert response.status == 200
+ assert response.body == FILES[path]
+ assert response.headers == {
+ b"Local Filename": [b""],
+ b"Size": [str(len(FILES[path])).encode()],
+ }
+ assert response.protocol == "FTP"
+ assert response.ip_address == ip_address("127.0.0.1")
+ assert response.certificate is None
+
+
+@coroutine_test
+async def test_download_ftps(
+ ftps_server: MockFTPServer, caplog: pytest.LogCaptureFixture
+) -> None:
+ settings = {"DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING": True}
+ with caplog.at_level(logging.DEBUG):
+ response = await _download(Request(ftps_server.url("file.txt")), settings)
+ assert response.body == FILES["file.txt"]
+ assert isinstance(response.certificate, bytes)
+ assert "SSL connection to 127.0.0.1 using protocol" in caplog.text
+
+
+@coroutine_test
+async def test_tls_verbose_logging_without_tls(ftp_server: MockFTPServer) -> None:
+ settings = {"DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING": True}
+ response = await _download(Request(ftp_server.url("file.txt")), settings)
+ assert response.body == FILES["file.txt"]
+
+
+@coroutine_test
+async def test_nonexistent(ftp_server: MockFTPServer) -> None:
+ response = await _download(Request(ftp_server.url("nonexistent.txt")))
+ assert response.status == 404
+ assert response.body == b""
+
+
[email protected]("path", ["", "/"])
+@coroutine_test
+async def test_list(ftp_server: MockFTPServer, path: str) -> None:
+ url = f"ftp://{ftp_server.host}:{ftp_server.port}{path}"
+ response = await _download(Request(url))
+ assert isinstance(response, TextResponse)
+ assert response.status == 200
+ assert set(response.text.splitlines()) == set(FILES)
+
+
+@coroutine_test
+async def test_list_nonexistent(ftp_server: MockFTPServer) -> None:
+ response = await _download(Request(ftp_server.url("nonexistent/")))
+ assert response.status == 404
+ assert response.body == b""
+
+
+@coroutine_test
+async def test_invalid_credentials(ftp_server: MockFTPServer) -> None:
+ request = Request(ftp_server.url("file.txt"), meta={"ftp_user": "invalid"})
+ with pytest.raises(DownloadFailedError):
+ await _download(request)
+
+
[email protected]("local", [False, True])
+@coroutine_test
+async def test_maxsize(ftp_server: MockFTPServer, tmp_path: Path, local: bool) -> None:
+ meta: dict[str, Any] = {"download_maxsize": 10}
+ if local:
+ meta["ftp_local_filename"] = str(tmp_path / "file.txt").encode()
+ request = Request(ftp_server.url("file.txt"), meta=meta)
+ with pytest.raises(DownloadCancelledError):
+ await _download(request)
+
+
+@coroutine_test
+async def test_maxsize_disabled(ftp_server: MockFTPServer) -> None:
+ request = Request(ftp_server.url("file.txt"), meta={"download_maxsize": 0})
+ response = await _download(request, {"DOWNLOAD_MAXSIZE": 10})
+ assert response.body == FILES["file.txt"]
+
+
+@coroutine_test
+async def test_warnsize(
+ ftp_server: MockFTPServer, caplog: pytest.LogCaptureFixture
+) -> None:
+ request = Request(ftp_server.url("file.txt"), meta={"download_warnsize": 10})
+ response = await _download(request)
+ assert response.body == FILES["file.txt"]
+ assert "which is larger than download warn size (10)" in caplog.text
+
+
+@coroutine_test
+async def test_active_mode_meta(
+ ftp_server: MockFTPServer, caplog: pytest.LogCaptureFixture
+) -> None:
+ request = Request(ftp_server.url("file.txt"), meta={"ftp_passive": False})
+ response = await _download(request)
+ assert response.body == FILES["file.txt"]
+ assert "does not support active mode" in caplog.text
+
+
+@coroutine_test
+async def test_active_mode() -> None:
+ crawler = get_crawler(settings_dict={"FTP_PASSIVE_MODE": False})
+ with pytest.raises(NotConfigured):
+ build_from_crawler(AioftpDownloadHandler, crawler)
+
+
+@coroutine_test
+async def test_connection_refused() -> None:
+ server = await asyncio.start_server(lambda r, w: None, "127.0.0.1", 0)
+ port = server.sockets[0].getsockname()[1]
+ server.close()
+ await server.wait_closed()
+ with pytest.raises(DownloadConnectionRefusedError):
+ await _download(Request(f"ftp://127.0.0.1:{port}/file.txt"))
+
+
+@coroutine_test
+async def test_cannot_resolve_host() -> None:
+ with pytest.raises(CannotResolveHostError):
+ await _download(Request("ftp://nonexistent.invalid/file.txt"))
+
+
+@coroutine_test
+async def test_timeout() -> None:
+ async def never_greet(
+ reader: asyncio.StreamReader, writer: asyncio.StreamWriter
+ ) -> None:
+ await reader.read()
+ writer.close()
+
+ async with _serve(never_greet) as url:
+ request = Request(url, meta={"download_timeout": 0.1})
+ with pytest.raises(DownloadTimeoutError):
+ await _download(request)
+
+
+@coroutine_test
+async def test_connection_lost() -> None:
+ async with _serve(lambda r, w: w.close()) as url:
+ with pytest.raises(DownloadFailedError):
+ await _download(Request(url))
+
+
+def _fake_ftp(
+ retr_reply: bytes, data: bytes = b""
+) -> Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any]:
+ """Return a minimal FTP server connection handler that sends *data*, if
+ any, and then answers RETR with *retr_reply*."""
+
+ async def handle(
+ reader: asyncio.StreamReader, writer: asyncio.StreamWriter
+ ) -> None:
+ data_writer: asyncio.Future[asyncio.StreamWriter] = (
+ asyncio.get_running_loop().create_future()
+ )
+
+ def on_data_connection(
+ _: asyncio.StreamReader, w: asyncio.StreamWriter
+ ) -> None:
+ data_writer.set_result(w)
+
+ writer.write(b"220 Ready\r\n")
+ while line := await reader.readline():
+ command = line.split()[0].upper()
+ if command == b"USER":
+ writer.write(b"230 Logged in\r\n")
+ elif command == b"TYPE":
+ writer.write(b"200 OK\r\n")
+ elif command == b"EPSV":
+ data_server = await asyncio.start_server(
+ on_data_connection, "127.0.0.1", 0
+ )
+ port = data_server.sockets[0].getsockname()[1]
+ writer.write(f"229 Entering passive mode (|||{port}|)\r\n".encode())
+ elif command == b"RETR":
+ w = await data_writer
+ if data:
+ writer.write(b"150 Sending\r\n")
+ w.write(data)
+ w.close()
+ data_server.close()
+ writer.write(retr_reply)
+ await writer.drain()
+ writer.close()
+
+ return handle
+
+
+@asynccontextmanager
+async def _serve(
+ handler: Callable[[asyncio.StreamReader, asyncio.StreamWriter], Any],
+) -> AsyncGenerator[str]:
+ server = await asyncio.start_server(handler, "127.0.0.1", 0)
+ port = server.sockets[0].getsockname()[1]
+ try:
+ yield f"ftp://127.0.0.1:{port}/file.txt"
+ finally:
+ server.close()
+
+
+@coroutine_test
+async def test_retr_failure() -> None:
+ async with _serve(_fake_ftp(b"450 Busy\r\n")) as url:
+ response = await _download(Request(url))
+ assert response.status == 503
+
+
+_TRUNCATED = _fake_ftp(b"426 Transfer aborted\r\n", data=b"partial")
+
+
+@coroutine_test
+async def test_dataloss() -> None:
+ async with _serve(_TRUNCATED) as url:
+ with pytest.raises(ResponseDataLossError):
+ await _download(Request(url))
+
+
+@coroutine_test
+async def test_dataloss_allowed() -> None:
+ async with _serve(_TRUNCATED) as url:
+ response = await _download(
+ Request(url, meta={"download_fail_on_dataloss": False})
+ )
+ assert response.body == b"partial"
+ assert response.flags == ["dataloss"]
+
+
+@coroutine_test
+async def test_local_filename(ftp_server: MockFTPServer, tmp_path: Path) -> None:
+ local_filename = str(tmp_path / "file.txt").encode()
+ request = Request(
+ ftp_server.url("file.txt"), meta={"ftp_local_filename": local_filename}
+ )
+ response = await _download(request)
+ assert response.body == local_filename
+ assert response.headers == {
+ b"Local Filename": [local_filename],
+ b"Size": [b"17"],
+ }
+ assert (tmp_path / "file.txt").read_bytes() == FILES["file.txt"]
+
+
+@asynccontextmanager
+async def _socks_proxy(**kwargs: Any) -> AsyncGenerator[str]:
+ server = await asyncio.start_server(
+ lambda r, w: socks_server_handler(r, w, **kwargs), "127.0.0.1", 0
+ )
+ port = server.sockets[0].getsockname()[1]
+ try:
+ yield f"127.0.0.1:{port}"
+ finally:
+ server.close()
+
+
+requires_socks = pytest.mark.skipif(
+ not _HAS_SOCKS or socks_server_handler is None,
+ reason="Requires python-socks and siosocks",
+)
+
+
+@requires_socks
[email protected]("scheme", ["socks4", "socks5"])
+@coroutine_test
+async def test_socks_proxy(ftp_server: MockFTPServer, scheme: str) -> None:
+ async with _socks_proxy() as proxy:
+ request = Request(
+ ftp_server.url("file.txt"), meta={"proxy": f"{scheme}://{proxy}"}
+ )
+ response = await _download(request)
+ assert response.body == FILES["file.txt"]
+
+
+@requires_socks
+@coroutine_test
+async def test_socks_proxy_credentials(ftp_server: MockFTPServer) -> None:
+ async with _socks_proxy(
+ allowed_versions={5}, username="user", password="pass"
+ ) as proxy:
+ request = Request(
+ ftp_server.url("file.txt"),
+ meta={"proxy": f"socks5://{proxy}"},
+ headers={"Proxy-Authorization": basic_auth_header("user", "pass")},
+ )
+ response = await _download(request)
+ assert response.body == FILES["file.txt"]
+
+
+@requires_socks
+@coroutine_test
+async def test_socks_proxy_wrong_credentials(ftp_server: MockFTPServer) -> None:
+ async with _socks_proxy(
+ allowed_versions={5}, username="user", password="pass"
+ ) as proxy:
+ request = Request(
+ ftp_server.url("file.txt"),
+ meta={"proxy": f"socks5://{proxy}"},
+ headers={"Proxy-Authorization": basic_auth_header("user", "wrong")},
+ )
+ with pytest.raises(DownloadFailedError):
+ await _download(request)
+
+
+@requires_socks
+@coroutine_test
+async def test_socks_proxy_ftps(ftps_server: MockFTPServer) -> None:
+ async with _socks_proxy() as proxy:
+ request = Request(
+ ftps_server.url("file.txt"), meta={"proxy": f"socks5://{proxy}"}
+ )
+ response = await _download(request)
+ assert response.body == FILES["file.txt"]
+ assert isinstance(response.certificate, bytes)
+
+
+@coroutine_test
+async def test_http_proxy(ftp_server: MockFTPServer) -> None:
+ request = Request(ftp_server.url("file.txt"), meta={"proxy": "http://127.0.0.1:1"})
+ with pytest.raises(NotImplementedError):
+ await _download(request)
diff --git a/tests/test_downloader_handler_twisted_ftp.py b/tests/test_downloader_handler_twisted_ftp.py
index 1f00b9497..0d4d42901 100644
--- a/tests/test_downloader_handler_twisted_ftp.py
+++ b/tests/test_downloader_handler_twisted_ftp.py
@@ -14,7 +14,7 @@ from twisted.internet.protocol import Factory, Protocol
from scrapy import Spider
from scrapy.core.downloader.handlers.ftp import FTPDownloadHandler
from scrapy.crawler import Crawler
-from scrapy.exceptions import DownloadFailedError, NotConfigured
+from scrapy.exceptions import DownloadCancelledError, DownloadFailedError, NotConfigured
from scrapy.http import HtmlResponse, Request, Response
from scrapy.http.response.text import TextResponse
from scrapy.utils.defer import deferred_f_from_coro_f, maybe_deferred_to_future
@@ -39,6 +39,7 @@ class TestFTPBase(ABC):
("file.txt", b"I have the power!"),
("file with spaces.txt", b"Moooooooooo power!"),
("html-file-without-extension", b"<!DOCTYPE html>\n<title>.</title>"),
+ ("big.bin", b"x" * 2**20),
)
@abstractmethod
@@ -144,6 +145,50 @@ class TestFTPBase(ABC):
assert local_path.read_bytes() == b"I have the power!"
local_path.unlink()
+ @pytest.mark.parametrize("filename", ["file.txt", "big.bin"])
+ @pytest.mark.parametrize("local", [False, True])
+ @deferred_f_from_coro_f
+ async def test_maxsize(
+ self,
+ server_url: str,
+ dh: FTPDownloadHandler,
+ tmp_path: Path,
+ filename: str,
+ local: bool,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ meta = {**self.req_meta, "download_maxsize": 10}
+ if local:
+ meta["ftp_local_filename"] = str(tmp_path / "local.txt").encode()
+ request = Request(url=server_url + filename, meta=meta)
+ with pytest.raises(DownloadCancelledError):
+ await dh.download_request(request)
+ assert "which is larger than download max size (10)" in caplog.text
+
+ @deferred_f_from_coro_f
+ async def test_maxsize_disabled(self, server_url: str, tmp_path: Path) -> None:
+ crawler = get_crawler(settings_dict={"DOWNLOAD_MAXSIZE": 10})
+ dh = build_from_crawler(FTPDownloadHandler, crawler)
+ meta = {**self.req_meta, "download_maxsize": 0}
+ request = Request(url=server_url + "file.txt", meta=meta)
+ r = await dh.download_request(request)
+ assert r.body == b"I have the power!"
+
+ @pytest.mark.parametrize("filename", ["file.txt", "big.bin"])
+ @deferred_f_from_coro_f
+ async def test_warnsize(
+ self,
+ server_url: str,
+ dh: FTPDownloadHandler,
+ filename: str,
+ caplog: pytest.LogCaptureFixture,
+ ) -> None:
+ meta = {**self.req_meta, "download_warnsize": 10}
+ request = Request(url=server_url + filename, meta=meta)
+ r = await dh.download_request(request)
+ assert r.status == 200
+ assert caplog.text.count("which is larger than download warn size (10)") == 1
+
@pytest.mark.parametrize(
("filename", "response_class"),
[
diff --git a/tests/test_downloadermiddleware_httpcompression.py b/tests/test_downloadermiddleware_httpcompression.py
index d96978a6a..bf5a64099 100644
--- a/tests/test_downloadermiddleware_httpcompression.py
+++ b/tests/test_downloadermiddleware_httpcompression.py
@@ -814,6 +814,26 @@ class TestHttpCompression:
),
]
+ @coroutine_test
+ async def test_download_warnsize_setting_ignored_with_request_maxsize(
+ self, caplog: pytest.LogCaptureFixture
+ ) -> None:
+ settings = {"DOWNLOAD_WARNSIZE": 10_000_000}
+ crawler = get_crawler(Spider, settings_dict=settings)
+ spider = crawler._create_spider("scrapytest.org")
+ mw = build_from_crawler(HttpCompressionMiddleware, crawler)
+ mw.open_spider(spider)
+ response = self._getresponse("bomb-gzip")
+ response.meta["download_maxsize"] = 20_000_000
+
+ assert response.request
+ caplog.clear()
+ with caplog.at_level(
+ WARNING, logger="scrapy.downloadermiddlewares.httpcompression"
+ ):
+ await mw.process_response(response.request, response)
+ assert not caplog.records
+
@coroutine_test
async def test_download_warnsize_request_meta_br(
self, caplog: pytest.LogCaptureFixture
diff --git a/tests/test_spider_sitemap.py b/tests/test_spider_sitemap.py
index 562e0337a..6f50e4547 100644
--- a/tests/test_spider_sitemap.py
+++ b/tests/test_spider_sitemap.py
@@ -498,6 +498,23 @@ Sitemap: /sitemap-relative-url.xml
),
]
+ def test_download_warnsize_setting_ignored_with_request_maxsize(
+ self, caplog: pytest.LogCaptureFixture
+ ) -> None:
+ settings = {"DOWNLOAD_WARNSIZE": 10_000_000}
+ crawler = get_crawler(settings_dict=settings)
+ spider = self.spider_class.from_crawler(crawler, "example.com")
+ body_path = Path(tests_datadir, "compressed", "bomb-gzip.bin")
+ body = body_path.read_bytes()
+ request = Request(
+ url="https://example.com", meta={"download_maxsize": 20_000_000}
+ )
+ response = Response(url="https://example.com", body=body, request=request)
+ caplog.clear()
+ with caplog.at_level(WARNING, logger="scrapy.spiders.sitemap"):
+ spider._get_sitemap_body(response)
+ assert not caplog.records
+
def test_download_warnsize_request_meta(
self, caplog: pytest.LogCaptureFixture
) -> None:
diff --git a/tests/utils/bases/download_handlers_http.py b/tests/utils/bases/download_handlers_http.py
index 6a3abd0ae..e1158cac8 100644
--- a/tests/utils/bases/download_handlers_http.py
+++ b/tests/utils/bases/download_handlers_http.py
@@ -787,6 +787,31 @@ class TestHttpBase(ABC):
)
assert caplog.text.count("download warn size (4)") == 1
+ @pytest.mark.parametrize(
+ ("meta", "warns"),
+ [
+ pytest.param({"download_maxsize": 100}, False, id="maxsize-only"),
+ pytest.param(
+ {"download_maxsize": 100, "download_warnsize": 4},
+ True,
+ id="maxsize-and-warnsize",
+ ),
+ ],
+ )
+ @coroutine_test
+ async def test_download_warnsize_with_request_maxsize(
+ self,
+ caplog: pytest.LogCaptureFixture,
+ mockserver: MockServer,
+ meta: dict[str, int],
+ warns: bool,
+ ) -> None:
+ request = Request(mockserver.url("/text", is_secure=self.is_secure), meta=meta)
+ async with self.get_dh({"DOWNLOAD_WARNSIZE": 4}) as download_handler:
+ response = await download_handler.download_request(request)
+ assert response.body == b"Works"
+ assert ("download warn size (4)" in caplog.text) is warns
+
@coroutine_test
async def test_download_with_warnsize_no_content_length(
self, caplog: pytest.LogCaptureFixture, mockserver: MockServer
diff --git a/tox.ini b/tox.ini
index cc7370b19..56326cb2e 100644
--- a/tox.ini
+++ b/tox.ini
@@ -74,6 +74,7 @@ deps =
Pillow==12.3.0
Protego==0.7.0
Twisted==26.4.0
+ aioftp==0.28.3
aiohttp==3.14.3
attrs==26.1.0
backports.zstd==1.7.0
@@ -88,6 +89,7 @@ deps =
ipython==9.17.1
pyOpenSSL==26.4.0
pytest==9.1.1
+ python-socks[asyncio]==3.1.1
socksio==1.0.0
types-Pygments==2.21.0.20260819
types-defusedxml==0.7.0.20260504
@@ -140,6 +142,7 @@ deps =
pytest==8.4.0
Protego==0.2.0
Twisted==21.7.0
+ aioftp==0.26.1
aiohttp==3.13.3
backports.zstd==1.3.0
brotli==1.2.0; implementation_name != "pypy"
@@ -183,6 +186,8 @@ deps =
{[testenv]deps}
Pillow
Twisted[http2]
+ python-socks[asyncio]
+ siosocks # SOCKS server for tests
boto3
bpython # optional for shell wrapper tests
colorlog
@@ -200,6 +205,8 @@ deps =
{[min]deps}
Pillow==9.2.0
Twisted[http2]==21.7.0
+ python-socks[asyncio]==2.6.0
+ siosocks==0.2.0 # SOCKS server for tests
boto3==1.35.2
bpython==0.7.1
colorlog==6.6.0
@@ -236,6 +243,7 @@ commands_pre =
git+https://github.com/python-pillow/Pillow \
git+https://github.com/MagicStack/uvloop \
git+https://github.com/aio-libs/aiohttp \
+ git+https://github.com/aio-libs/aioftp \
git+https://github.com/pyca/cryptography \
git+https://github.com/scrapy/cssselect \
git+https://github.com/tiran/defusedxml \
@@ -324,6 +332,7 @@ basepython = pypy3.11
deps =
PyPyDispatcher==2.1.0
{[test-requirements]deps}
+ aioftp==0.26.1
aiohttp==3.13.3
pytest==8.4.0
Protego==0.2.0