Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Docs for HttpxDownloadHandler, handle httpx not being installed (#7368)

Andrey Rakhmatullin31bf7c3892df
diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst
index 7e672f0b4..54a7f0e7c 100644
--- a/docs/topics/download-handlers.rst
+++ b/docs/topics/download-handlers.rst
@@ -206,6 +206,57 @@ uses the HTTP/1.1 protocol for them.
It's implemented using :mod:`twisted.web.client`.
+HttpxDownloadHandler
+--------------------
+
+.. autoclass:: scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler
+
+| Supported schemes: ``http``, ``https``.
+| Lazy: no.
+
+This handler supports ``http://host/path`` and ``https://host/path`` URLs and
+uses the HTTP/1.1 protocol for them.
+
+It's implemented using the ``httpx`` library and needs it to be installed.
+
+If you want to use this handler you need to replace the default ones for the
+``http`` and ``https`` schemes:
+
+.. code-block:: python
+
+ DOWNLOAD_HANDLERS = {
+ "http": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+ "https": "scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler",
+ }
+
+.. warning::
+
+ This handler is experimental, and not yet recommended for production
+ environments. Future Scrapy versions may introduce related changes without
+ a deprecation period or warning or even remove it altogether.
+
+.. note::
+
+ As this handler is based on a different HTTP client implementation compared
+ to :class:`~.HTTP11DownloadHandler`, it's expected that its behavior on
+ some websites may be different. Additionally, these are the Scrapy features
+ that are explicitly not supported when using it:
+
+ - Proxy support (the :reqmeta:`proxy` meta key).
+
+ - Per-request bind address support (the :reqmeta:`bindaddress` meta key).
+ The global :setting:`DOWNLOAD_BIND_ADDRESS` setting is supported but the
+ port number, if specified, will be ignored.
+
+ - The :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
+ :setting:`DOWNLOADER_CLIENT_TLS_METHOD` settings.
+
+ - Settings specific to the Twisted networking or HTTP implementation, like
+ :setting:`DNS_RESOLVER`.
+
+ - Using :ref:`non-asyncio reactors <disable-asyncio>` (``httpx`` requires
+ ``asyncio``).
+
S3DownloadHandler
-----------------
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 1a4f91d93..64722f0e3 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -726,7 +726,7 @@ HttpProxyMiddleware
.. class:: HttpProxyMiddleware
This middleware sets the HTTP proxy to use for requests, by setting the
- ``proxy`` meta value for :class:`~scrapy.Request` objects.
+ :reqmeta:`proxy` meta value for :class:`~scrapy.Request` objects.
Like the Python standard library module :mod:`urllib.request`, it obeys
the following environment variables:
@@ -735,11 +735,18 @@ HttpProxyMiddleware
* ``https_proxy``
* ``no_proxy``
- You can also set the meta key ``proxy`` per-request, to a value like
+ You can also set the meta key :reqmeta:`proxy` per-request, to a value like
``http://some_proxy_server:port`` or ``http://username:password@some_proxy_server:port``.
Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
environment variables, and it will also ignore ``no_proxy`` environment variable.
+.. note::
+
+ Handling of this meta key needs to be implemented inside the :ref:`download
+ handler <topics-download-handlers>`, so it's not guaranteed to be supported
+ by all 3rd-party handlers. It's currently unsupported by
+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
+
HttpProxyMiddleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 5acabb0de..f23c7611c 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -646,7 +646,7 @@ bindaddress
The default local outgoing address for download-handler connections.
-This setting can be either:
+This meta value can be either:
- a host address as a string (e.g. ``"127.0.0.2"``), in which case the local
port is chosen automatically, or
@@ -675,6 +675,10 @@ If not set, built-in HTTP download handlers use the value of
Set the :reqmeta:`bindaddress` request meta key to override it for a
specific request.
+This meta key is not supported by
+:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`, but the
+:setting:`DOWNLOAD_BIND_ADDRESS` is supported by it.
+
.. reqmeta:: download_timeout
download_timeout
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 6e8d0b54f..c004c253e 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -714,12 +714,6 @@ or even enable client-side authentication (and various other things).
``'scrapy.core.downloader.contextfactory.BrowserLikeContextFactory'``,
which uses the platform's certificates to validate remote endpoints.
-If you do use a custom ContextFactory, make sure its ``__init__`` method
-accepts a ``method`` parameter (this is the ``OpenSSL.SSL`` method mapping
-:setting:`DOWNLOADER_CLIENT_TLS_METHOD`), a ``tls_verbose_logging``
-parameter (``bool``) and a ``tls_ciphers`` parameter (see
-:setting:`DOWNLOADER_CLIENT_TLS_CIPHERS`).
-
.. note::
This setting is specific to the built-in Twisted-based download handlers:
@@ -753,6 +747,8 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it.
(:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
:class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs
to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.
+ It's currently unsupported by
+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
.. setting:: DOWNLOADER_CLIENT_TLS_METHOD
@@ -783,6 +779,8 @@ This setting must be one of these string values:
(:class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
:class:`scrapy.core.downloader.handlers.http2.H2DownloadHandler`) it needs
to be implemented in the :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY` class.
+ It's currently unsupported by
+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
.. setting:: DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING
@@ -930,6 +928,13 @@ If set, built-in HTTP download handlers use this value by default.
Set the :reqmeta:`bindaddress` request meta key to override it for a specific
request.
+.. note::
+
+ Handling of this setting needs to be implemented inside the :ref:`download
+ handler <topics-download-handlers>`, so it's not guaranteed to be supported
+ by all 3rd-party handlers. Specifying the port is unsupported by
+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.
+
.. setting:: DOWNLOAD_HANDLERS
DOWNLOAD_HANDLERS
diff --git a/scrapy/core/downloader/handlers/_httpx.py b/scrapy/core/downloader/handlers/_httpx.py
index 916937152..b19a66696 100644
--- a/scrapy/core/downloader/handlers/_httpx.py
+++ b/scrapy/core/downloader/handlers/_httpx.py
@@ -9,8 +9,6 @@ from http.cookiejar import Cookie, CookieJar
from io import BytesIO
from typing import TYPE_CHECKING, Any, NoReturn, TypedDict
-import httpx
-
from scrapy import Request, signals
from scrapy.exceptions import (
CannotResolveHostError,
@@ -46,6 +44,11 @@ if TYPE_CHECKING:
from scrapy.crawler import Crawler
+try:
+ import httpx
+except ImportError:
+ httpx = None # type: ignore[assignment]
+
logger = logging.getLogger(__name__)
@@ -81,6 +84,10 @@ class HttpxDownloadHandler(BaseHttpDownloadHandler):
f" TWISTED_ENABLED setting. See the asyncio documentation"
f" of Scrapy for more information."
)
+ if httpx is None: # pragma: no cover
+ raise NotConfigured(
+ f"{type(self).__name__} requires the httpx library to be installed."
+ )
super().__init__(crawler)
logger.warning(
"HttpxDownloadHandler is experimental and is not recommented for production use."