Scrapy, a fast high-level web crawling & scraping framework for Python.
Docs for HttpxDownloadHandler, handle httpx not being installed (#7368)
Andrey Rakhmatullin31bf7c3892df
diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex 7e672f0b4..54a7f0e7c 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -206,6 +206,57 @@ uses the HTTP/1.1 protocol for them.It's implemented using :mod:`twisted.web.client`.+HttpxDownloadHandler+--------------------++.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler++| Supported schemes: ``http``, ``https``.+| Lazy: no.++This handler supports ``http://host/path`` and ``https://host/path`` URLs and+uses the HTTP/1.1 protocol for them.++It's implemented using the ``httpx`` library and needs it to be installed.++If you want to use this handler you need to replace the default ones for the+``http`` and ``https`` schemes:++.. code-block:: python++ DOWNLOAD_HANDLERS = {+ "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",+ "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",+ }++.. warning::++ This handler is experimental, and not yet recommended for production+ environments. Future Scrapy versions may introduce related changes without+ a deprecation period or warning or even remove it altogether.++.. note::++ As this handler is based on a different HTTP client implementation compared+ to :class:`~.HTTP11DownloadHandler`, it's expected that its behavior on+ some websites may be different. Additionally, these are the Scrapy features+ that are explicitly not supported when using it:++ - Proxy support (the :reqmeta:`proxy` meta key).++ - Per-request bind address support (the :reqmeta:`bindaddress` meta key).+ The global :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported but the+ port number, if specified, will be ignored.++ - The :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and+ :setting:`DOWNLOADER_CLIENT_TLS_METHOD` settings.++ - Settings specific to the Twisted networking or HTTP implementation, like+ :setting:`DNS_RESOLVER`.++ - Using :ref:`non-asyncio reactors <disable-asyncio>` (``httpx`` requires+ ``asyncio``).+S3DownloadHandler-----------------diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 1a4f91d93..64722f0e3 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -726,7 +726,7 @@ HttpProxyMiddleware.. class:: HttpProxyMiddlewareThis middleware sets the HTTP proxy to use for requests, by setting the- ``proxy`` meta value for :class:`~scrapy.Request` objects.+ :reqmeta:`proxy` meta value for :class:`~scrapy.Request` objects.Like the Python standard library module :mod:`urllib.request`, it obeysthe following environment variables:@@ -735,11 +735,18 @@ HttpProxyMiddleware* ``https_proxy``* ``no_proxy``- You can also set the meta key ``proxy`` per-request, to a value like+ You can also set the meta key :reqmeta:`proxy` per-request, to a value like``http://some_proxy_server:port`` or ``http://username:password@some_proxy_server:port``.Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``environment variables, and it will also ignore ``no_proxy`` environment variable.+.. note::++ Handling of this meta key needs to be implemented inside the :ref:`download+ handler <topics-download-handlers>`, so it's not guaranteed to be supported+ by all 3rd-party handlers. It's currently unsupported by+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.+HttpProxyMiddleware settings~~~~~~~~~~~~~~~~~~~~~~~~~~~~diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 5acabb0de..f23c7611c 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -646,7 +646,7 @@ bindaddressThe default local outgoing address for download-handler connections.-This setting can be either:+This meta value can be either:- a host address as a string (e.g. ``"127.0.0.2"``), in which case the localport is chosen automatically, or@@ -675,6 +675,10 @@ If not set, built-in HTTP download handlers use the value ofSet the :reqmeta:`bindaddress` request meta key to override it for aspecific request.+This meta key is not supported by+:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the+:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it.+.. reqmeta:: download_timeoutdownload_timeoutdiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 6e8d0b54f..c004c253e 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -714,12 +714,6 @@ or even enable client-side authentication (and various other things).``'scrapy.core.downloader.contextfactory.BrowserLikeContextFactory'``,which uses the platform's certificates to validate remote endpoints.-If you do use a custom ContextFactory, make sure its ``__init__`` method-accepts a ``method`` parameter (this is the ``OpenSSL.SSL`` method mapping-:setting:`DOWNLOADER_CLIENT_TLS_METHOD`), a ``tls_verbose_logging``-parameter (``bool``) and a ``tls_ciphers`` parameter (see-:setting:`DOWNLOADER_CLIENT_TLS_CIPHERS`).-.. note::This setting is specific to the built-in Twisted-based download handlers:@@ -753,6 +747,8 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it.(:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and:class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needsto be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.+ It's currently unsupported by+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`... setting:: DOWNLOADER_CLIENT_TLS_METHOD@@ -783,6 +779,8 @@ This setting must be one of these string values:(:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and:class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needsto be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.+ It's currently unsupported by+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`... setting:: DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING@@ -930,6 +928,13 @@ If set, built-in HTTP download handlers use this value by default.Set the :reqmeta:`bindaddress` request meta key to override it for a specificrequest.+.. note::++ Handling of this setting needs to be implemented inside the :ref:`download+ handler <topics-download-handlers>`, so it's not guaranteed to be supported+ by all 3rd-party handlers. Specifying the port is unsupported by+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.+.. setting:: DOWNLOAD_HANDLERSDOWNLOAD_HANDLERSdiff --git a/scrapy/core/downloader/handlers/_httpx.py b/scrapy/core/downloader/handlers/_httpx.pyindex 916937152..b19a66696 100644--- a/scrapy/core/downloader/handlers/_httpx.py+++ b/scrapy/core/downloader/handlers/_httpx.py@@ -9,8 +9,6 @@ from http.cookiejar import Cookie, CookieJarfrom io import BytesIOfrom typing import TYPE_CHECKING, Any, NoReturn, TypedDict-import httpx-from scrapy import Request, signalsfrom scrapy.exceptions import (CannotResolveHostError,@@ -46,6 +44,11 @@ if TYPE_CHECKING:from scrapy.crawler import Crawler+try:+ import httpx+except ImportError:+ httpx = None # type: ignore[assignment]+logger = logging.getLogger(__name__)@@ -81,6 +84,10 @@ class HttpxDownloadHandler(BaseHttpDownloadHandler):f" TWISTED_ENABLED setting. See the asyncio documentation"f" of Scrapy for more information.")+ if httpx is None: # pragma: no cover+ raise NotConfigured(+ f"{type(self).__name__} requires the httpx library to be installed."+ )super().__init__(crawler)logger.warning("HttpxDownloadHandler is experimental and is not recommented for production use."