Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add AiohttpDownloadHandler and switch reactorless to it (#8118)

Andrey Rakhmatullin41c3f3e1ffba
diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index afccb491d..e3258b037 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -152,9 +152,7 @@ Using Scrapy without a Twisted reactor
.. note:: As the Twisted download handlers cannot be used without a reactor,
the default download handler in this mode is
- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. You
- will need to additionally install the :ref:`httpx <extras>` extra to use
- it, unless you switch to some different handler.
+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`.
It's possible to use Scrapy without installing a Twisted reactor at all, by
setting the :setting:`TWISTED_REACTOR_ENABLED` setting to ``False``. In this
@@ -191,7 +189,7 @@ in future Scrapy versions. The following features are not available:
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` (this
is likely the biggest difference; Scrapy provides an HTTP(S) download handler
that doesn't require a reactor and will be used instead of it:
- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`)
+ :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`)
* :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler`
* :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`
* :ref:`topics-telnetconsole`
@@ -217,12 +215,12 @@ the defaults of some other settings:
* :setting:`TELNETCONSOLE_ENABLED` is set to ``False``.
* The ``"http"`` and ``"https"`` keys in :setting:`DOWNLOAD_HANDLERS_BASE` are
- set to ``"scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler"``.
+ set to ``"scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"``.
* The ``"ftp"`` key in :setting:`DOWNLOAD_HANDLERS_BASE` is set to ``None``.
-Thus, :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` is
-used by default for making HTTP(S) requests. Please refer to its documentation
-for its differences and limitations compared to
+Thus, :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
+is used by default for making HTTP(S) requests. Please refer to its
+documentation for its differences and limitations compared to
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.
Additionally, :class:`~scrapy.crawler.AsyncCrawlerProcess` will install a
diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst
index dc24680aa..6f9f95a70 100644
--- a/docs/topics/download-handlers.rst
+++ b/docs/topics/download-handlers.rst
@@ -130,18 +130,40 @@ using different handlers.
Here is a comparison of some features of the built-in HTTP handlers, see the
individual handler docs for more differences:
-=================== ================= ===================== ====================
-Feature H2DownloadHandler HTTP11DownloadHandler HttpxDownloadHandler
-=================== ================= ===================== ====================
-Requires asyncio No No Yes
-Requires a reactor Yes Yes No
-HTTP/1.1 No Yes Yes
-HTTP/2 Yes No Yes
-TLS implementation ``cryptography`` ``cryptography`` Stdlib ``ssl``
-HTTP proxies No Yes Yes
-SOCKS proxies No No Yes
-Bad header handling Not applicable Skip bad Fail
-=================== ================= ===================== ====================
+.. list-table::
+ :header-rows: 1
+ :stub-columns: 1
+
+ * - Handler
+ - Requirements
+ - HTTP
+ - Proxies
+ - Bad headers
+ - TLS
+ * - :class:`Aiohttp <scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler>`
+ - asyncio
+ - 1.1
+ - HTTP
+ - Fail
+ - Stdlib ``ssl``
+ * - :class:`H2 <scrapy.core.downloader.handlers.http2.H2DownloadHandler>`
+ - Reactor, :ref:`twisted-http2 <extras>` extra
+ - 2
+ - None
+ - Not applicable
+ - ``cryptography``
+ * - :class:`HTTP11 <scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler>`
+ - Reactor
+ - 1.1
+ - HTTP
+ - Skip bad
+ - ``cryptography``
+ * - :class:`Httpx <scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler>`
+ - asyncio, :ref:`httpx <extras>` extra
+ - 1.1, 2
+ - HTTP, SOCKS
+ - Fail
+ - Stdlib ``ssl``
Bad header handling is what a handler does when a response has a bad header
line, e.g. one with no colon in it, which some servers send. Handlers that skip
@@ -156,6 +178,60 @@ of this package for more information.
.. _scrapy-download-handlers-incubator: https://github.com/scrapy-plugins/scrapy-download-handlers-incubator
+.. _aiohttp-handler:
+
+AiohttpDownloadHandler
+----------------------
+
+.. versionadded:: VERSION
+
+.. autoclass:: scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler
+
+| Supported schemes: ``http``, ``https``.
+| :ref:`Lazy <lazy-download-handlers>`: no.
+| :ref:`Requires asyncio support <using-asyncio>`: yes.
+| :ref:`Requires a Twisted reactor <asyncio-without-reactor>`: no.
+
+This handler supports ``http://host/path`` and ``https://host/path`` URLs and
+uses the HTTP/1.1 protocol for them.
+
+It's implemented using the aiohttp_ library.
+
+.. _aiohttp: https://docs.aiohttp.org/
+
+If you want to use this handler you need to replace the default ones for the
+``http`` and ``https`` schemes:
+
+.. code-block:: python
+
+ DOWNLOAD_HANDLERS = {
+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ }
+
+Features and limitations
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning::
+
+ This handler is experimental, and not yet recommended for production
+ environments. Future Scrapy versions may introduce related changes without
+ a deprecation period or warning or even remove it altogether.
+
+=========================== =======================================
+HTTP proxies Yes
+SOCKS proxies No (not supported by the library)
+HTTP/2 No (not supported by the library)
+Bad header handling Fail (not supported by the library)
+``response.certificate`` DER bytes
+Per-request ``bindaddress`` No (not supported by the library)
+TLS implementation Standard library ``ssl``
+=========================== =======================================
+
+Other limitations:
+
+- HTTPS proxies for HTTPS destinations are not supported on Python < 3.11.
+
.. _twisted-http2-handler:
H2DownloadHandler
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 93dcc9ab4..d001c46b9 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -909,8 +909,9 @@ Set the :reqmeta:`bindaddress` request meta key to override it for a
specific request.
This meta key is not supported by
-:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the
-:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it.
+:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` and
+:class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`, but
+the :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported by them.
.. reqmeta:: download_timeout
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 77e8585be..71e27ad86 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1080,8 +1080,8 @@ Default:
{
"data": "scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler",
"file": "scrapy.core.downloader.handlers.file.FileDownloadHandler",
- "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
- "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
"s3": "scrapy.core.downloader.handlers.s3.S3DownloadHandler",
"ftp": None,
}
diff --git a/pyproject.toml b/pyproject.toml
index 51e29e72e..9b44e649f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -311,6 +311,9 @@ filterwarnings = [
"ignore:unclosed <socket\\.socket fd=:ResourceWarning",
"ignore:.*<function (_SelectorTransport|_ProactorBasePipeTransport|_ProactorSocketTransport|BaseEventLoop)\\.__del__:pytest.PytestUnraisableExceptionWarning",
"ignore:unclosed transport <:ResourceWarning",
+ # These are triggered by garbage collecting an unclosed AiohttpDownloadHandler.
+ "ignore:.*<function (ClientSession|BaseConnector)\\.__del__:pytest.PytestUnraisableExceptionWarning",
+ "ignore:Unclosed (client session|connector) <aiohttp\\.:ResourceWarning",
# PyDispatcher iterates its connection registry while a weakref callback
# may mutate it, which raises RuntimeError on PyPy, where several weakrefs
# can die within a single collection.
diff --git a/scrapy/core/downloader/handlers/_aiohttp.py b/scrapy/core/downloader/handlers/_aiohttp.py
new file mode 100644
index 000000000..75be971dd
--- /dev/null
+++ b/scrapy/core/downloader/handlers/_aiohttp.py
@@ -0,0 +1,160 @@
+from __future__ import annotations
+
+import asyncio
+import ipaddress
+import ssl
+from contextlib import asynccontextmanager
+from typing import TYPE_CHECKING, ClassVar, cast
+
+import aiohttp
+import aiohttp.connector
+import yarl
+
+from scrapy.exceptions import (
+ CannotResolveHostError,
+ DownloadConnectionRefusedError,
+ DownloadFailedError,
+ DownloadTimeoutError,
+ UnsupportedURLSchemeError,
+)
+from scrapy.http import Headers
+from scrapy.utils.ssl import _log_sslobj_debug_info, _make_ssl_context
+
+from ._base_streaming import BaseStreamingDownloadHandler, _BaseResponseArgs
+
+if TYPE_CHECKING:
+ from collections.abc import AsyncIterator
+
+ from scrapy import Request
+ from scrapy.crawler import Crawler
+
+
+class _ClientResponse(aiohttp.ClientResponse):
+ """Captures transport data that can be lost after parent ``start()``.
+
+ Workaround for https://github.com/aio-libs/aiohttp/issues/2205.
+ """
+
+ _peername: tuple[str, int] | None = None
+ _ssl_object: ssl.SSLObject | None = None
+
+ async def start(
+ self, connection: aiohttp.connector.Connection
+ ) -> aiohttp.ClientResponse:
+ transport = connection.transport
+ assert transport is not None
+ self._peername = transport.get_extra_info("peername")
+ ssl_object = transport.get_extra_info("ssl_object")
+ if isinstance(ssl_object, ssl.SSLObject):
+ self._ssl_object = ssl_object
+ return await super().start(connection)
+
+
+class AiohttpDownloadHandler(BaseStreamingDownloadHandler[_ClientResponse]):
+ experimental: ClassVar[bool] = True
+
+ def __init__(self, crawler: Crawler):
+ super().__init__(crawler)
+ self._ssl_context: ssl.SSLContext = _make_ssl_context(crawler.settings)
+ connector = aiohttp.TCPConnector(
+ local_addr=self._bind_address,
+ # hard limit on simultaneous connections
+ limit=self._pool_size_total,
+ # hard limit on simultaneous connections per host
+ limit_per_host=self._pool_size_per_host,
+ )
+ self._session: aiohttp.ClientSession = aiohttp.ClientSession(
+ connector=connector,
+ cookie_jar=aiohttp.DummyCookieJar(),
+ auto_decompress=False,
+ response_class=_ClientResponse,
+ skip_auto_headers=(
+ "Accept",
+ "Accept-Encoding",
+ "Content-Type",
+ "User-Agent",
+ ),
+ )
+
+ @asynccontextmanager
+ async def _make_request(
+ self, request: Request, timeout: float
+ ) -> AsyncIterator[_ClientResponse]:
+ proxy = self._extract_proxy_url_with_creds(request)
+ headers = self._request_headers(request).to_tuple_list()
+ url: str | yarl.URL = request.url
+ if request.meta.get("verbatim_url"):
+ # encoded=True disables the percent-encoding normalization that
+ # yarl applies to str URLs, so the URL is sent as is
+ url = yarl.URL(request.url, encoded=True)
+ try:
+ async with await self._session.request(
+ request.method,
+ url,
+ data=request.body,
+ headers=headers,
+ timeout=aiohttp.ClientTimeout(total=timeout),
+ ssl=self._ssl_context,
+ allow_redirects=False,
+ proxy=proxy,
+ ) as response:
+ yield cast("_ClientResponse", response)
+ except (TimeoutError, asyncio.TimeoutError) as e:
+ raise DownloadTimeoutError(
+ f"Getting {request.url} took longer than {timeout} seconds."
+ ) from e
+ except (
+ aiohttp.InvalidUrlClientError,
+ aiohttp.NonHttpUrlClientError,
+ ) as e:
+ raise UnsupportedURLSchemeError(str(e)) from e
+ except aiohttp.ClientConnectorDNSError as e:
+ raise CannotResolveHostError(str(e)) from e
+ except aiohttp.ClientConnectorError as e:
+ raise DownloadConnectionRefusedError(str(e)) from e
+ except aiohttp.ClientError as e:
+ raise DownloadFailedError(str(e)) from e
+
+ @staticmethod
+ def _extract_headers(response: _ClientResponse) -> Headers:
+ return Headers(list(response.headers.items()))
+
+ @staticmethod
+ def _build_base_response_args(
+ response: _ClientResponse,
+ request: Request,
+ headers: Headers,
+ ) -> _BaseResponseArgs:
+ version = response.version
+ protocol_version = (
+ f"HTTP/{version.major}.{version.minor}" if version else "HTTP/1.1"
+ )
+ assert response._peername is not None
+ ip_address = ipaddress.ip_address(response._peername[0])
+ if response._ssl_object is not None:
+ cert = response._ssl_object.getpeercert(binary_form=True)
+ else: # HTTP
+ cert = None
+ return {
+ "status": response.status,
+ "url": request.url,
+ "headers": headers,
+ "certificate": cert,
+ "ip_address": ip_address,
+ "protocol": protocol_version,
+ }
+
+ def _log_tls_info(self, response: _ClientResponse, request: Request) -> None:
+ assert response._ssl_object is not None
+ _log_sslobj_debug_info(response._ssl_object)
+
+ @staticmethod
+ def _iter_body_chunks(response: _ClientResponse) -> AsyncIterator[bytes]:
+ return response.content.iter_any()
+
+ @staticmethod
+ def _is_dataloss_exception(exc: Exception) -> bool:
+ return isinstance(exc, aiohttp.ClientPayloadError)
+
+ async def close(self) -> None:
+ await self._session.close()
diff --git a/scrapy/core/downloader/handlers/_base_streaming.py b/scrapy/core/downloader/handlers/_base_streaming.py
index 5a669c732..6e4dd6870 100644
--- a/scrapy/core/downloader/handlers/_base_streaming.py
+++ b/scrapy/core/downloader/handlers/_base_streaming.py
@@ -90,12 +90,6 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon
"CONCURRENT_REQUESTS_PER_DOMAIN"
)
- @staticmethod
- @abstractmethod
- def _check_deps_installed() -> None:
- """Raise NotConfigured if the required deps are not installed."""
- raise NotImplementedError
-
@abstractmethod
def _make_request(
self, request: Request, timeout: float
@@ -133,6 +127,10 @@ class BaseStreamingDownloadHandler(BaseHttpDownloadHandler, ABC, Generic[_Respon
"""Return True if ``exc`` represents dataloss."""
raise NotImplementedError
+ @staticmethod
+ def _check_deps_installed() -> None:
+ """Raise NotConfigured if the required deps are not installed."""
+
def _log_tls_info(self, response: _ResponseT, request: Request) -> None:
"""Log TLS connection details, if possible."""
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index 554cccc6e..7d07c8711 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -249,7 +249,7 @@ class Crawler:
self.settings.set("TELNETCONSOLE_ENABLED", False, priority="default")
for scheme in ("http", "https"):
self.settings["DOWNLOAD_HANDLERS_BASE"][scheme] = (
- "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler"
+ "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler"
)
self.settings["DOWNLOAD_HANDLERS_BASE"]["ftp"] = None
diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py
index 7619d23b2..f9e4d3989 100644
--- a/scrapy/utils/test.py
+++ b/scrapy/utils/test.py
@@ -51,8 +51,8 @@ def get_reactor_settings() -> dict[str, Any]:
settings["TWISTED_REACTOR_ENABLED"] = False
settings["DOWNLOAD_HANDLERS"] = {
"ftp": None,
- "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
- "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
}
return settings
diff --git a/tests/test_core_downloader.py b/tests/test_core_downloader.py
index 67734a688..034a4b034 100644
--- a/tests/test_core_downloader.py
+++ b/tests/test_core_downloader.py
@@ -638,6 +638,7 @@ async def test_slot_gc() -> None:
assert not downloader.slots
finally:
downloader.close()
+ await downloader.handlers._close()
@coroutine_test
diff --git a/tests/test_crawler_subprocess.py b/tests/test_crawler_subprocess.py
index cbb6f2c5b..d2b6f6b6a 100644
--- a/tests/test_crawler_subprocess.py
+++ b/tests/test_crawler_subprocess.py
@@ -369,7 +369,7 @@ class TestAsyncCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
assert "Spider closed (finished)" in log
assert "is_reactorless(): True" in log
assert "ERROR: " not in log
- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2
+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2
assert log.count("WARNING: ") == 2
def test_reactorless_custom_settings(self) -> None:
@@ -391,7 +391,7 @@ class TestAsyncCrawlerProcessSubprocess(TestCrawlerProcessSubprocessBase):
assert "{'data': 'foo'}" in log
assert "'item_scraped_count': 1" in log
assert "ERROR: " not in log
- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2
+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2
assert log.count("WARNING: ") == 2
def test_reactorless_import_hook(self) -> None:
@@ -604,7 +604,7 @@ class TestAsyncCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):
assert "Spider closed (finished)" in log
assert "is_reactorless(): True" in log
assert "ERROR: " not in log
- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2
+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2
assert log.count("WARNING: ") == 2
def test_reactorless_custom_settings(self) -> None:
@@ -623,7 +623,7 @@ class TestAsyncCrawlerRunnerSubprocess(TestCrawlerRunnerSubprocessBase):
assert "{'data': 'foo'}" in log
assert "'item_scraped_count': 1" in log
assert "ERROR: " not in log
- assert log.count("WARNING: HttpxDownloadHandler is experimental") == 2
+ assert log.count("WARNING: AiohttpDownloadHandler is experimental") == 2
assert log.count("WARNING: ") == 2
def test_reactorless_reactor(self) -> None:
diff --git a/tests/test_downloader_handler_aiohttp.py b/tests/test_downloader_handler_aiohttp.py
new file mode 100644
index 000000000..99bb763c2
--- /dev/null
+++ b/tests/test_downloader_handler_aiohttp.py
@@ -0,0 +1,122 @@
+from __future__ import annotations
+
+import sys
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from scrapy.core.downloader.handlers._aiohttp import AiohttpDownloadHandler
+from tests.utils.bases.download_handlers_http import (
+ TestHttpBase,
+ TestHttpProxyBase,
+ TestHttpsBase,
+ TestHttpsCustomCiphersBase,
+ TestHttpsDefaultCiphersBase,
+ TestHttpsInvalidDNSIdBase,
+ TestHttpsInvalidDNSPatternBase,
+ TestHttpsTLSVersionBase,
+ TestHttpsWrongHostnameBase,
+ TestHttpWithCrawlerBase,
+ TestMitmProxyBase,
+ TestRealWebsiteBase,
+ TestSimpleHttpsBase,
+)
+
+if TYPE_CHECKING:
+ from scrapy.core.downloader.handlers import DownloadHandlerProtocol
+
+pytestmark = pytest.mark.only_asyncio
+
+
+class AiohttpDownloadHandlerMixin:
+ @property
+ def download_handler_cls(self) -> type[DownloadHandlerProtocol]:
+ return AiohttpDownloadHandler
+
+ @property
+ def settings_dict(self) -> dict[str, Any] | None:
+ return {
+ "DOWNLOAD_HANDLERS": {
+ "http": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ "https": "scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler",
+ }
+ }
+
+
+class TestHttp(AiohttpDownloadHandlerMixin, TestHttpBase):
+ handler_supports_bindaddress_meta = False
+ handler_bad_header_handling = "fail"
+
+
+class TestHttps(AiohttpDownloadHandlerMixin, TestHttpsBase):
+ handler_supports_bindaddress_meta = False
+ handler_bad_header_handling = "fail"
+ tls_log_message = "SSL connection to 127.0.0.1 using protocol TLSv1.3, cipher"
+
+ @pytest.mark.skip(reason="The check is Twisted-specific")
+ def test_verify_certs_deprecated(self) -> None: # type: ignore[override]
+ pass
+
+
+class TestSimpleHttps(AiohttpDownloadHandlerMixin, TestSimpleHttpsBase):
+ pass
+
+
+class TestHttpsWrongHostname(AiohttpDownloadHandlerMixin, TestHttpsWrongHostnameBase):
+ pass
+
+
+class TestHttpsInvalidDNSId(AiohttpDownloadHandlerMixin, TestHttpsInvalidDNSIdBase):
+ pass
+
+
+class TestHttpsInvalidDNSPattern(
+ AiohttpDownloadHandlerMixin, TestHttpsInvalidDNSPatternBase
+):
+ pass
+
+
+class TestHttpsCustomCiphers(AiohttpDownloadHandlerMixin, TestHttpsCustomCiphersBase):
+ pass
+
+
+class TestHttpsDefaultCiphers(AiohttpDownloadHandlerMixin, TestHttpsDefaultCiphersBase):
+ pass
+
+
+class TestHttpsTLSVersion(AiohttpDownloadHandlerMixin, TestHttpsTLSVersionBase):
+ pass
+
+
+class TestHttpWithCrawler(AiohttpDownloadHandlerMixin, TestHttpWithCrawlerBase):
+ pass
+
+
+class TestHttpsWithCrawler(TestHttpWithCrawler):
+ is_secure = True
+
+
+class TestHttpProxy(AiohttpDownloadHandlerMixin, TestHttpProxyBase):
+ pass
+
+
+class TestHttpsProxy(AiohttpDownloadHandlerMixin, TestHttpProxyBase):
+ is_secure = True
+
+ @property
+ def handler_supports_tls_in_tls(self) -> bool:
+ return sys.version_info >= (3, 11)
+
+
+class TestMitmProxy(AiohttpDownloadHandlerMixin, TestMitmProxyBase):
+ handler_supports_socks: bool = False
+
+ @property
+ def handler_supports_tls_in_tls(self) -> bool:
+ return sys.version_info >= (3, 11)
+
+
+class TestRealWebsite(AiohttpDownloadHandlerMixin, TestRealWebsiteBase):
+ pass
diff --git a/tests/test_downloaderslotssettings.py b/tests/test_downloaderslotssettings.py
index 1c373a3bd..f2eaabfa6 100644
--- a/tests/test_downloaderslotssettings.py
+++ b/tests/test_downloaderslotssettings.py
@@ -102,6 +102,7 @@ async def test_params():
request = Request("https://example.com")
_, actual = downloader._get_slot(request)
downloader.close()
+ await downloader.handlers._close()
expected = Slot(**params)
for param in params:
assert getattr(expected, param) == getattr(actual, param), (
@@ -122,6 +123,7 @@ async def test_deprecated_randomize_delay_param(value: bool, expected: float):
):
_, slot = downloader._get_slot(Request("https://example.com"))
downloader.close()
+ await downloader.handlers._close()
assert slot.jitter == expected
@@ -139,6 +141,7 @@ async def test_get_slot_deprecated_spider_arg():
key1, slot1 = downloader._get_slot(request, spider=crawler.spider)
key2, slot2 = downloader._get_slot(request)
downloader.close()
+ await downloader.handlers._close()
assert key1 == key2
assert slot1 == slot2
diff --git a/tests/utils/bases/download_handlers_http.py b/tests/utils/bases/download_handlers_http.py
index 7b54ed21f..914b8ce80 100644
--- a/tests/utils/bases/download_handlers_http.py
+++ b/tests/utils/bases/download_handlers_http.py
@@ -1498,14 +1498,17 @@ class TestHttpWithCrawlerBase(ABC):
class TestHttpProxyBase(ABC):
is_secure = False
expected_http_proxy_request_body = b"http://example.com"
- # whether the handler supports HTTPS proxies with HTTPS destinations
- handler_supports_tls_in_tls: bool = True
@property
@abstractmethod
def download_handler_cls(self) -> type[DownloadHandlerProtocol]:
raise NotImplementedError
+ # whether the handler supports HTTPS proxies with HTTPS destinations
+ @property
+ def handler_supports_tls_in_tls(self) -> bool:
+ return True
+
@pytest.fixture(scope="session")
def proxy_mockserver(self) -> Generator[ProxyEchoMockServer]:
with ProxyEchoMockServer() as proxy:
@@ -1579,8 +1582,6 @@ PROXY_KINDS = ["http", "https", "socks5"]
class TestMitmProxyBase(ABC):
- # whether the handler supports HTTPS proxies with HTTPS destinations
- handler_supports_tls_in_tls: bool = True
handler_supports_socks: bool = False
@property
@@ -1588,6 +1589,11 @@ class TestMitmProxyBase(ABC):
def settings_dict(self) -> dict[str, Any] | None:
raise NotImplementedError
+ # whether the handler supports HTTPS proxies with HTTPS destinations
+ @property
+ def handler_supports_tls_in_tls(self) -> bool:
+ return True
+
def _maybe_skip(self, proxy_kind: str, https_dest: bool) -> None:
if proxy_kind == "socks5" and not self.handler_supports_socks:
pytest.skip("SOCKS proxies are not supported")
diff --git a/tox.ini b/tox.ini
index bea257c7b..dfbb110a6 100644
--- a/tox.ini
+++ b/tox.ini
@@ -38,7 +38,6 @@ minversion = 1.7.0
deps =
attrs
coverage >= 7.10.6
- httpx2
pexpect >= 4.8.0
psutil
pyftpdlib >= 2.0.1
@@ -149,7 +148,6 @@ deps =
charset-normalizer==3.4.0
cryptography==41.0.5
cssselect==0.9.1
- httpx2==2.0.0
itemadapter==0.1.0
lxml==4.6.4
parsel==1.5.0
@@ -233,6 +231,7 @@ commands_pre =
git+https://github.com/twisted/twisted \
git+https://github.com/python-pillow/Pillow \
git+https://github.com/MagicStack/uvloop \
+ git+https://github.com/aio-libs/aiohttp \
git+https://github.com/pyca/cryptography \
git+https://github.com/scrapy/cssselect \
git+https://github.com/tiran/defusedxml \
@@ -330,7 +329,6 @@ deps =
charset-normalizer==3.4.0
cryptography==44.0.2
cssselect==0.9.1
- httpx2==2.0.0
itemadapter==0.1.0
lxml==5.3.2
parsel==1.5.0