Scrapy, a fast high-level web crawling & scraping framework for Python.
Add AiohttpDownloadHandler and switch reactorless to it (#8118)
Andrey Rakhmatullin41c3f3e1ffba
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex afccb491d..e3258b037 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -152,9 +152,7 @@ Using Scrapy without a Twisted reactor.. note:: As the Twisted download handlers cannot be used without a reactor,the default download handler in this mode is- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. You- will need to additionally install the :ref:`httpx <extras>` extra to use- it, unless you switch to some different handler.+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`.It's possible to use Scrapy without installing a Twisted reactor at all, bysetting the :setting:`TWISTED_REACTOR_ENABLED` setting to ``False``. In this@@ -191,7 +189,7 @@ in future Scrapy versions. The following features are not available::class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` (thisis likely the biggest difference; Scrapy provides an HTTP(S) download handlerthat doesn't require a reactor and will be used instead of it:- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`)+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`)* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`* :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`* :ref:`topics-telnetconsole`@@ -217,12 +215,12 @@ the defaults of some other settings:* :setting:`TELNETCONSOLE_ENABLED` is set to ``False``.* The ``"http"`` and ``"https"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` are- set to ``"scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler"``.+ set to ``"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"``.* The ``"ftp"`` key in :setting:`DOWNLOAD_HANDLERS_BASE` is set to ``None``.-Thus, :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` is-used by default for making HTTP(S) requests. Please refer to its documentation-for its differences and limitations compared to+Thus, :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`+is used by default for making HTTP(S) requests. Please refer to its+documentation for its differences and limitations compared to:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.Additionally, :class:`~scrapy.crawler.AsyncCrawlerProcess` will install adiff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex dc24680aa..6f9f95a70 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -130,18 +130,40 @@ using different handlers.Here is a comparison of some features of the built-in HTTP handlers, see theindividual handler docs for more differences:-=================== ================= ===================== ====================-Feature H2DownloadHandler HTTP11DownloadHandler HttpxDownloadHandler-=================== ================= ===================== ====================-Requires asyncio No No Yes-Requires a reactor Yes Yes No-HTTP/1.1 No Yes Yes-HTTP/2 Yes No Yes-TLS implementation ``cryptography`` ``cryptography`` Stdlib ``ssl``-HTTP proxies No Yes Yes-SOCKS proxies No No Yes-Bad header handling Not applicable Skip bad Fail-=================== ================= ===================== ====================+.. list-table::+ :header-rows: 1+ :stub-columns: 1++ * - Handler+ - Requirements+ - HTTP+ - Proxies+ - Bad headers+ - TLS+ * - :class:`Aiohttp <scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler>`+ - asyncio+ - 1.1+ - HTTP+ - Fail+ - Stdlib ``ssl``+ * - :class:`H2 <scrapy.core.downloader.handlers.http2.H2DownloadHandler>`+ - Reactor, :ref:`twisted-http2 <extras>` extra+ - 2+ - None+ - Not applicable+ - ``cryptography``+ * - :class:`HTTP11 <scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler>`+ - Reactor+ - 1.1+ - HTTP+ - Skip bad+ - ``cryptography``+ * - :class:`Httpx <scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler>`+ - asyncio, :ref:`httpx <extras>` extra+ - 1.1, 2+ - HTTP, SOCKS+ - Fail+ - Stdlib ``ssl``Bad header handling is what a handler does when a response has a bad headerline, e.g. one with no colon in it, which some servers send. Handlers that skip@@ -156,6 +178,60 @@ of this package for more information... _scrapy-download-handlers-incubator: https://github.com/scrapy-plugins/scrapy-download-handlers-incubator+.. _aiohttp-handler:++AiohttpDownloadHandler+----------------------++.. versionadded:: VERSION++.. autoclass:: scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler++| Supported schemes: ``http``, ``https``.+| :ref:`Lazy <lazy-download-handlers>`: no.+| :ref:`Requires asyncio support <using-asyncio>`: yes.+| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.++This handler supports ``http://host/path`` and ``https://host/path`` URLs and+uses the HTTP/1.1 protocol for them.++It's implemented using the aiohttp_ library.++.. _aiohttp: https://docs.aiohttp.org/++If you want to use this handler you need to replace the default ones for the+``http`` and ``https`` schemes:++.. code-block:: python++ DOWNLOAD_HANDLERS = {+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ }++Features and limitations+^^^^^^^^^^^^^^^^^^^^^^^^++.. warning::++ This handler is experimental, and not yet recommended for production+ environments. Future Scrapy versions may introduce related changes without+ a deprecation period or warning or even remove it altogether.++=========================== =======================================+HTTP proxies Yes+SOCKS proxies No (not supported by the library)+HTTP/2 No (not supported by the library)+Bad header handling Fail (not supported by the library)+``response.certificate`` DER bytes+Per-request ``bindaddress`` No (not supported by the library)+TLS implementation Standard library ``ssl``+=========================== =======================================++Other limitations:++- HTTPS proxies for HTTPS destinations are not supported on Python < 3.11.+.. _twisted-http2-handler:H2DownloadHandlerdiff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 93dcc9ab4..d001c46b9 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -909,8 +909,9 @@ Set the :reqmeta:`bindaddress` request meta key to override it for aspecific request.This meta key is not supported by-:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the-:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it.+:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` and+:class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`, but+the :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported by them... reqmeta:: download_timeoutdiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 77e8585be..71e27ad86 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1080,8 +1080,8 @@ Default:{"data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler","file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",- "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",- "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler","s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler","ftp": None,}diff --git a/pyproject.toml b/pyproject.tomlindex 51e29e72e..9b44e649f 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -311,6 +311,9 @@ filterwarnings = ["ignore:unclosed <socket\\.socket fd=:ResourceWarning","ignore:.*<function (_SelectorTransport|_ProactorBasePipeTransport|_ProactorSocketTransport|BaseEventLoop)\\.__del__:pytest.PytestUnraisableExceptionWarning","ignore:unclosed transport <:ResourceWarning",+ # These are triggered by garbage collecting an unclosed AiohttpDownloadHandler.+ "ignore:.*<function (ClientSession|BaseConnector)\\.__del__:pytest.PytestUnraisableExceptionWarning",+ "ignore:Unclosed (client session|connector) <aiohttp\\.:ResourceWarning",# PyDispatcher iterates its connection registry while a weakref callback# may mutate it, which raises RuntimeError on PyPy, where several weakrefs# can die within a single collection.diff --git a/scrapy/core/downloader/handlers/_aiohttp.py b/scrapy/core/downloader/handlers/_aiohttp.pynew file mode 100644index 000000000..75be971dd--- /dev/null+++ b/scrapy/core/downloader/handlers/_aiohttp.py@@ -0,0 +1,160 @@+from __future__ import annotations++import asyncio+import ipaddress+import ssl+from contextlib import asynccontextmanager+from typing import TYPE_CHECKING, ClassVar, cast++import aiohttp+import aiohttp.connector+import yarl++from scrapy.exceptions import (+ CannotResolveHostError,+ DownloadConnectionRefusedError,+ DownloadFailedError,+ DownloadTimeoutError,+ UnsupportedURLSchemeError,+)+from scrapy.http import Headers+from scrapy.utils.ssl import _log_sslobj_debug_info, _make_ssl_context++from ._base_streaming import BaseStreamingDownloadHandler, _BaseResponseArgs++if TYPE_CHECKING:+ from collections.abc import AsyncIterator++ from scrapy import Request+ from scrapy.crawler import Crawler+++class _ClientResponse(aiohttp.ClientResponse):+ """Captures transport data that can be lost after parent ``start()``.++ Workaround for https://github.com/aio-libs/aiohttp/issues/2205.+ """++ _peername: tuple[str, int] | None = None+ _ssl_object: ssl.SSLObject | None = None++ async def start(+ self, connection: aiohttp.connector.Connection+ ) -> aiohttp.ClientResponse:+ transport = connection.transport+ assert transport is not None+ self._peername = transport.get_extra_info("peername")+ ssl_object = transport.get_extra_info("ssl_object")+ if isinstance(ssl_object, ssl.SSLObject):+ self._ssl_object = ssl_object+ return await super().start(connection)+++class AiohttpDownloadHandler(BaseStreamingDownloadHandler[_ClientResponse]):+ experimental: ClassVar[bool] = True++ def __init__(self, crawler: Crawler):+ super().__init__(crawler)+ self._ssl_context: ssl.SSLContext = _make_ssl_context(crawler.settings)+ connector = aiohttp.TCPConnector(+ local_addr=self._bind_address,+ # hard limit on simultaneous connections+ limit=self._pool_size_total,+ # hard limit on simultaneous connections per host+ limit_per_host=self._pool_size_per_host,+ )+ self._session: aiohttp.ClientSession = aiohttp.ClientSession(+ connector=connector,+ cookie_jar=aiohttp.DummyCookieJar(),+ auto_decompress=False,+ response_class=_ClientResponse,+ skip_auto_headers=(+ "Accept",+ "Accept-Encoding",+ "Content-Type",+ "User-Agent",+ ),+ )++ @asynccontextmanager+ async def _make_request(+ self, request: Request, timeout: float+ ) -> AsyncIterator[_ClientResponse]:+ proxy = self._extract_proxy_url_with_creds(request)+ headers = self._request_headers(request).to_tuple_list()+ url: str | yarl.URL = request.url+ if request.meta.get("verbatim_url"):+ # encoded=True disables the percent-encoding normalization that+ # yarl applies to str URLs, so the URL is sent as is+ url = yarl.URL(request.url, encoded=True)+ try:+ async with await self._session.request(+ request.method,+ url,+ data=request.body,+ headers=headers,+ timeout=aiohttp.ClientTimeout(total=timeout),+ ssl=self._ssl_context,+ allow_redirects=False,+ proxy=proxy,+ ) as response:+ yield cast("_ClientResponse", response)+ except (TimeoutError, asyncio.TimeoutError) as e:+ raise DownloadTimeoutError(+ f"Getting {request.url} took longer than {timeout} seconds."+ ) from e+ except (+ aiohttp.InvalidUrlClientError,+ aiohttp.NonHttpUrlClientError,+ ) as e:+ raise UnsupportedURLSchemeError(str(e)) from e+ except aiohttp.ClientConnectorDNSError as e:+ raise CannotResolveHostError(str(e)) from e+ except aiohttp.ClientConnectorError as e:+ raise DownloadConnectionRefusedError(str(e)) from e+ except aiohttp.ClientError as e:+ raise DownloadFailedError(str(e)) from e++ @staticmethod+ def _extract_headers(response: _ClientResponse) -> Headers:+ return Headers(list(response.headers.items()))++ @staticmethod+ def _build_base_response_args(+ response: _ClientResponse,+ request: Request,+ headers: Headers,+ ) -> _BaseResponseArgs:+ version = response.version+ protocol_version = (+ f"HTTP/{version.major}.{version.minor}" if version else "HTTP/1.1"+ )+ assert response._peername is not None+ ip_address = ipaddress.ip_address(response._peername[0])+ if response._ssl_object is not None:+ cert = response._ssl_object.getpeercert(binary_form=True)+ else: # HTTP+ cert = None+ return {+ "status": response.status,+ "url": request.url,+ "headers": headers,+ "certificate": cert,+ "ip_address": ip_address,+ "protocol": protocol_version,+ }++ def _log_tls_info(self, response: _ClientResponse, request: Request) -> None:+ assert response._ssl_object is not None+ _log_sslobj_debug_info(response._ssl_object)++ @staticmethod+ def _iter_body_chunks(response: _ClientResponse) -> AsyncIterator[bytes]:+ return response.content.iter_any()++ @staticmethod+ def _is_dataloss_exception(exc: Exception) -> bool:+ return isinstance(exc, aiohttp.ClientPayloadError)++ async def close(self) -> None:+ await self._session.close()diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.pyindex 5a669c732..6e4dd6870 100644--- a/scrapy/core/downloader/handlers/_base_streaming.py+++ b/scrapy/core/downloader/handlers/_base_streaming.py@@ -90,12 +90,6 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon"CONCURRENT_REQUESTS_PER_DOMAIN")- @staticmethod- @abstractmethod- def _check_deps_installed() -> None:- """Raise NotConfigured if the required deps are not installed."""- raise NotImplementedError-@abstractmethoddef _make_request(self, request: Request, timeout: float@@ -133,6 +127,10 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon"""Return True if ``exc`` represents dataloss."""raise NotImplementedError+ @staticmethod+ def _check_deps_installed() -> None:+ """Raise NotConfigured if the required deps are not installed."""+def _log_tls_info(self, response: _ResponseT, request: Request) -> None:"""Log TLS connection details, if possible."""diff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex 554cccc6e..7d07c8711 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -249,7 +249,7 @@ class Crawler:self.settings.set("TELNETCONSOLE_ENABLED", False, priority="default")for scheme in ("http", "https"):self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = (- "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler"+ "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler")self.settings["DOWNLOAD_HANDLERS_BASE"]["ftp"] = Nonediff --git a/scrapy/utils/test.py b/scrapy/utils/test.pyindex 7619d23b2..f9e4d3989 100644--- a/scrapy/utils/test.py+++ b/scrapy/utils/test.py@@ -51,8 +51,8 @@ def get_reactor_settings() -> dict[str, Any]:settings["TWISTED_REACTOR_ENABLED"] = Falsesettings["DOWNLOAD_HANDLERS"] = {"ftp": None,- "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",- "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",}return settingsdiff --git a/tests/test_core_downloader.py b/tests/test_core_downloader.pyindex 67734a688..034a4b034 100644--- a/tests/test_core_downloader.py+++ b/tests/test_core_downloader.py@@ -638,6 +638,7 @@ async def test_slot_gc() -> None:assert not downloader.slotsfinally:downloader.close()+ await downloader.handlers._close()@coroutine_testdiff --git a/tests/test_crawler_subprocess.py b/tests/test_crawler_subprocess.pyindex cbb6f2c5b..d2b6f6b6a 100644--- a/tests/test_crawler_subprocess.py+++ b/tests/test_crawler_subprocess.py@@ -369,7 +369,7 @@ class TestAsyncCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):assert "Spider closed (finished)" in logassert "is_reactorless(): True" in logassert "ERROR: " not in log- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2assert log.count("WARNING: ") == 2def test_reactorless_custom_settings(self) -> None:@@ -391,7 +391,7 @@ class TestAsyncCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):assert "{'data': 'foo'}" in logassert "'item_scraped_count': 1" in logassert "ERROR: " not in log- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2assert log.count("WARNING: ") == 2def test_reactorless_import_hook(self) -> None:@@ -604,7 +604,7 @@ class TestAsyncCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):assert "Spider closed (finished)" in logassert "is_reactorless(): True" in logassert "ERROR: " not in log- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2assert log.count("WARNING: ") == 2def test_reactorless_custom_settings(self) -> None:@@ -623,7 +623,7 @@ class TestAsyncCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):assert "{'data': 'foo'}" in logassert "'item_scraped_count': 1" in logassert "ERROR: " not in log- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2assert log.count("WARNING: ") == 2def test_reactorless_reactor(self) -> None:diff --git a/tests/test_downloader_handler_aiohttp.py b/tests/test_downloader_handler_aiohttp.pynew file mode 100644index 000000000..99bb763c2--- /dev/null+++ b/tests/test_downloader_handler_aiohttp.py@@ -0,0 +1,122 @@+from __future__ import annotations++import sys+from typing import TYPE_CHECKING, Any++import pytest++from scrapy.core.downloader.handlers._aiohttp import AiohttpDownloadHandler+from tests.utils.bases.download_handlers_http import (+ TestHttpBase,+ TestHttpProxyBase,+ TestHttpsBase,+ TestHttpsCustomCiphersBase,+ TestHttpsDefaultCiphersBase,+ TestHttpsInvalidDNSIdBase,+ TestHttpsInvalidDNSPatternBase,+ TestHttpsTLSVersionBase,+ TestHttpsWrongHostnameBase,+ TestHttpWithCrawlerBase,+ TestMitmProxyBase,+ TestRealWebsiteBase,+ TestSimpleHttpsBase,+)++if TYPE_CHECKING:+ from scrapy.core.downloader.handlers import DownloadHandlerProtocol++pytestmark = pytest.mark.only_asyncio+++class AiohttpDownloadHandlerMixin:+ @property+ def download_handler_cls(self) -> type[DownloadHandlerProtocol]:+ return AiohttpDownloadHandler++ @property+ def settings_dict(self) -> dict[str, Any] | None:+ return {+ "DOWNLOAD_HANDLERS": {+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",+ }+ }+++class TestHttp(AiohttpDownloadHandlerMixin, TestHttpBase):+ handler_supports_bindaddress_meta = False+ handler_bad_header_handling = "fail"+++class TestHttps(AiohttpDownloadHandlerMixin, TestHttpsBase):+ handler_supports_bindaddress_meta = False+ handler_bad_header_handling = "fail"+ tls_log_message = "SSL connection to 127.0.0.1 using protocol TLSv1.3, cipher"++ @pytest.mark.skip(reason="The check is Twisted-specific")+ def test_verify_certs_deprecated(self) -> None: # type: ignore[override]+ pass+++class TestSimpleHttps(AiohttpDownloadHandlerMixin, TestSimpleHttpsBase):+ pass+++class TestHttpsWrongHostname(AiohttpDownloadHandlerMixin, TestHttpsWrongHostnameBase):+ pass+++class TestHttpsInvalidDNSId(AiohttpDownloadHandlerMixin, TestHttpsInvalidDNSIdBase):+ pass+++class TestHttpsInvalidDNSPattern(+ AiohttpDownloadHandlerMixin, TestHttpsInvalidDNSPatternBase+):+ pass+++class TestHttpsCustomCiphers(AiohttpDownloadHandlerMixin, TestHttpsCustomCiphersBase):+ pass+++class TestHttpsDefaultCiphers(AiohttpDownloadHandlerMixin, TestHttpsDefaultCiphersBase):+ pass+++class TestHttpsTLSVersion(AiohttpDownloadHandlerMixin, TestHttpsTLSVersionBase):+ pass+++class TestHttpWithCrawler(AiohttpDownloadHandlerMixin, TestHttpWithCrawlerBase):+ pass+++class TestHttpsWithCrawler(TestHttpWithCrawler):+ is_secure = True+++class TestHttpProxy(AiohttpDownloadHandlerMixin, TestHttpProxyBase):+ pass+++class TestHttpsProxy(AiohttpDownloadHandlerMixin, TestHttpProxyBase):+ is_secure = True++ @property+ def handler_supports_tls_in_tls(self) -> bool:+ return sys.version_info >= (3, 11)++[email protected]_mitmproxy+class TestMitmProxy(AiohttpDownloadHandlerMixin, TestMitmProxyBase):+ handler_supports_socks: bool = False++ @property+ def handler_supports_tls_in_tls(self) -> bool:+ return sys.version_info >= (3, 11)++[email protected]_internet+class TestRealWebsite(AiohttpDownloadHandlerMixin, TestRealWebsiteBase):+ passdiff --git a/tests/test_downloaderslotssettings.py b/tests/test_downloaderslotssettings.pyindex 1c373a3bd..f2eaabfa6 100644--- a/tests/test_downloaderslotssettings.py+++ b/tests/test_downloaderslotssettings.py@@ -102,6 +102,7 @@ async def test_params():request = Request("https://example.com")_, actual = downloader._get_slot(request)downloader.close()+ await downloader.handlers._close()expected = Slot(**params)for param in params:assert getattr(expected, param) == getattr(actual, param), (@@ -122,6 +123,7 @@ async def test_deprecated_randomize_delay_param(value: bool, expected: float):):_, slot = downloader._get_slot(Request("https://example.com"))downloader.close()+ await downloader.handlers._close()assert slot.jitter == expected@@ -139,6 +141,7 @@ async def test_get_slot_deprecated_spider_arg():key1, slot1 = downloader._get_slot(request, spider=crawler.spider)key2, slot2 = downloader._get_slot(request)downloader.close()+ await downloader.handlers._close()assert key1 == key2assert slot1 == slot2diff --git a/tests/utils/bases/download_handlers_http.py b/tests/utils/bases/download_handlers_http.pyindex 7b54ed21f..914b8ce80 100644--- a/tests/utils/bases/download_handlers_http.py+++ b/tests/utils/bases/download_handlers_http.py@@ -1498,14 +1498,17 @@ class TestHttpWithCrawlerBase(ABC):class TestHttpProxyBase(ABC):is_secure = Falseexpected_http_proxy_request_body = b"http://example.com"- # whether the handler supports HTTPS proxies with HTTPS destinations- handler_supports_tls_in_tls: bool = True@property@abstractmethoddef download_handler_cls(self) -> type[DownloadHandlerProtocol]:raise NotImplementedError+ # whether the handler supports HTTPS proxies with HTTPS destinations+ @property+ def handler_supports_tls_in_tls(self) -> bool:+ return True+@pytest.fixture(scope="session")def proxy_mockserver(self) -> Generator[ProxyEchoMockServer]:with ProxyEchoMockServer() as proxy:@@ -1579,8 +1582,6 @@ PROXY_KINDS = ["http", "https", "socks5"]class TestMitmProxyBase(ABC):- # whether the handler supports HTTPS proxies with HTTPS destinations- handler_supports_tls_in_tls: bool = Truehandler_supports_socks: bool = False@property@@ -1588,6 +1589,11 @@ class TestMitmProxyBase(ABC):def settings_dict(self) -> dict[str, Any] | None:raise NotImplementedError+ # whether the handler supports HTTPS proxies with HTTPS destinations+ @property+ def handler_supports_tls_in_tls(self) -> bool:+ return True+def _maybe_skip(self, proxy_kind: str, https_dest: bool) -> None:if proxy_kind == "socks5" and not self.handler_supports_socks:pytest.skip("SOCKS proxies are not supported")diff --git a/tox.ini b/tox.iniindex bea257c7b..dfbb110a6 100644--- a/tox.ini+++ b/tox.ini@@ -38,7 +38,6 @@ minversion = 1.7.0deps =attrscoverage >= 7.10.6- httpx2pexpect >= 4.8.0psutilpyftpdlib >= 2.0.1@@ -149,7 +148,6 @@ deps =charset-normalizer==3.4.0cryptography==41.0.5cssselect==0.9.1- httpx2==2.0.0itemadapter==0.1.0lxml==4.6.4parsel==1.5.0@@ -233,6 +231,7 @@ commands_pre =git+https://github.com/twisted/twisted \git+https://github.com/python-pillow/Pillow \git+https://github.com/MagicStack/uvloop \+ git+https://github.com/aio-libs/aiohttp \git+https://github.com/pyca/cryptography \git+https://github.com/scrapy/cssselect \git+https://github.com/tiran/defusedxml \@@ -330,7 +329,6 @@ deps =charset-normalizer==3.4.0cryptography==44.0.2cssselect==0.9.1- httpx2==2.0.0itemadapter==0.1.0lxml==5.3.2parsel==1.5.0