Scrapy, a fast high-level web crawling & scraping framework for Python.
Release notes for 2.17.0 (#7723)
Andrey Rakhmatullin5ccc8dbe8a73
diff --git a/docs/news.rst b/docs/news.rstindex b8b976df9..6966e7763 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,6 +3,310 @@Release notes=============+.. _release-2.17.0:++Scrapy 2.17.0 (unreleased)+--------------------------++Highlights:++- HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler``++- Improved settings for changing allowed TLS versions++Deprecations+~~~~~~~~~~~~++- The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use+ the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or+ :setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change+ the TLS method selection.+ (:issue:`3288`, :issue:`6546`)++- The following spider attributes are deprecated in favor of settings:++ - ``http_user`` (use :setting:`HTTPAUTH_USER`)++ - ``http_pass`` (use :setting:`HTTPAUTH_PASS`)++ - ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`)++ (:issue:`7590`)++- The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was+ never called by Scrapy.+ (:issue:`7626`, :issue:`7633`)++- The following TLS-related functions and constants, intended for internal+ use, are deprecated:++ - ``scrapy.core.downloader.tls.METHOD_TLS``++ - ``scrapy.core.downloader.tls.METHOD_TLSv10``++ - ``scrapy.core.downloader.tls.METHOD_TLSv11``++ - ``scrapy.core.downloader.tls.METHOD_TLSv12``++ - ``scrapy.core.downloader.tls.openssl_methods``++ - ``scrapy.core.downloader.tls.DEFAULT_CIPHERS``++ - ``scrapy.utils.ssl.ffi_buf_to_string()``++ - ``scrapy.utils.ssl.get_temp_key_info()``++ - ``scrapy.utils.ssl.x509name_to_string()``++ (:issue:`6546`, :issue:`7619`, :issue:`7665`)++- The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set+ ``follow=False`` in your rules to achieve the same effect.+ (:issue:`7592`)++- Instantiating+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`+ without a ``crawler`` argument is deprecated.+ (:issue:`7655`)++- Instantiating+ :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a+ ``settings`` argument is deprecated.+ (:issue:`7664`)++New features+~~~~~~~~~~~~++- Added support for HTTP/2 requests to+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It+ requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to+ ``True``.+ (:issue:`7575`)++- Added support for SOCKS proxies to+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.+ (:issue:`747`, :issue:`7575`)++- Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and+ :setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the+ ``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated).+ Compared to the old setting, they support specifying a range of allowed+ versions and support newer TLS versions.+ (:issue:`4821`, :issue:`6546`)++- Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and+ :setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`,+ :reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more+ flexible ways to set HTTP authentication data.+ (:issue:`7590`)++- Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to+ skip request URL canonicalization.+ (:issue:`7473`)++- Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`.+ (:issue:`6321`, :issue:`7679`)++- :attr:`scrapy.Item.fields` now returns the fields in the definition order+ instead of the alphabetical one.+ (:issue:`7015`, :issue:`7694`)++- Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a+ :reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument+ of the+ :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that+ allow changing the log level of the message logged when the retry limit has+ been reached.+ (:issue:`4622`, :issue:`5297`, :issue:`7567`)++- It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to+ ``None`` to use the default ciphers of the underlying TLS implementation.+ (:issue:`7499`, :issue:`7665`)++Improvements+~~~~~~~~~~~~++- :class:`~scrapy.FormRequest` is no longer deprecated, only its+ ``from_response()`` method is still deprecated.+ (:issue:`7561`, :issue:`7671`)++- Switched the item definition in the default project template from a+ :class:`scrapy.item.Item` to a dataclass.+ (:issue:`7493`, :issue:`7513`)++- Fixed deprecation warnings with pyOpenSSL 26.3.0.+ (:issue:`7619`)++- Removed the runtime warnings for :attr:`Spider.allowed_domains+ <scrapy.Spider.allowed_domains>` containing URLs or domains with ports+ instead of just domains and for spider classes having a ``start_url``+ attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use+ :doc:`scrapy-lint <scrapy-lint:index>` to find mistakes in your spider code+ instead.+ (:issue:`4421`, :issue:`7627`)++- :func:`scrapy.utils.test.get_crawler` now disables+ :setting:`TELNETCONSOLE_ENABLED` by default.+ (:issue:`7644`)++- Other code refactoring and improvements.+ (:issue:`7409`, :issue:`7593`, :issue:`7594`, :issue:`7611`, :issue:`7649`)++Bug fixes+~~~~~~~~~++- :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no+ longer ignores proxy credentials for redirected or retried requests.+ (:issue:`7601`, :issue:`7630`)++- :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the+ temporary file after the upload.+ (:issue:`7546`)++- Fixed ``scrapy shell <URL>`` running a full spider crawl when there is a+ spider for the requested URL. This bug was introduced in Scrapy 2.13.0.+ (:issue:`7552`, :issue:`7557`)++- The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL`+ settings are no longer ignored. This bug was introduced in Scrapy 2.12.0.+ (:issue:`7597`, :issue:`7614`)++- :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes+ the connection after making the request.+ (:issue:`7602`, :issue:`7667`)++- Removed the deprecated ``spider`` argument from the pipeline defined in the+ default project template.+ (:issue:`7676`)++- Fixed ``scrapy genspider --edit`` not working.+ (:issue:`7260`, :issue:`7683`)++- When a :class:`~scrapy.crawler.Crawler` instance is passed to+ :meth:`AsyncCrawlerRunner.create_crawler()+ <scrapy.crawler.AsyncCrawlerRunner.create_crawler>` or+ :meth:`CrawlerRunner.create_crawler()+ <scrapy.crawler.CrawlerRunner.create_crawler>`, settings from both classes+ are now merged, previously only the settings from the+ :class:`~scrapy.crawler.Crawler` instance were used.+ (:issue:`1280`, :issue:`7647`)++- Fixed several issues with cookie handling in+ :func:`scrapy.utils.request.request_to_curl`.+ (:issue:`7603`, :issue:`7675`, :issue:`7684`)++- Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the+ cache when :setting:`DNSCACHE_ENABLED` is set to ``False``.+ (:issue:`7663`)++- Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments+ when looking for the ``<base>`` tag.+ (:issue:`7506`)++- Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR`+ implementations.+ (:issue:`7589`)++- Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with+ credentials.+ (:issue:`7604`, :issue:`7605`)++- :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when+ looking for "nofollow" strings.+ (:issue:`7632`)++- Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing+ some malformed sitemaps.+ (:issue:`7686`, :issue:`7687`)++Documentation+~~~~~~~~~~~~~++- Mentioned :doc:`scrapy-lint <scrapy-lint:index>` in the docs.+ (:issue:`4421`, :issue:`7627`)++- Added the docs about :ref:`security considerations <security>`.+ (:issue:`7389`, :issue:`7678`)++- Improved the :ref:`item pipeline docs <topics-item-pipeline>`.+ (:issue:`2350`, :issue:`7676`)++- Documented which stats are collected by+ :class:`~scrapy.extensions.corestats.CoreStats`.+ (:issue:`7421`)++- Switched documentation examples from using :class:`scrapy.item.Item` to+ using dataclasses.+ (:issue:`7493`, :issue:`7513`)++- Added feature comparison tables to the :ref:`download handler+ <download-handlers-ref>` docs.+ (:issue:`7575`)++- Improved the docs for :ref:`logging settings <logging-settings>`.+ (:issue:`6909`, :issue:`7668`)++- Documented a way to :ref:`improve startup time and memory usage+ <large-project-startup>` by using :setting:`SPIDER_MODULES`.+ (:issue:`7576`, :issue:`7600`)++- Clarified handling of the ``type`` argument of :class:`~scrapy.Selector`.+ (:issue:`7704`)++- Other documentation improvements and fixes.+ (:issue:`4954`,+ :issue:`6120`,+ :issue:`7286`,+ :issue:`7564`,+ :issue:`7573`,+ :issue:`7598`,+ :issue:`7599`,+ :issue:`7698`)++Quality assurance+~~~~~~~~~~~~~~~~~++- Fixed deprecation warnings with pytest 9.1.0.+ (:issue:`7621`)++- Type hints improvements and fixes.+ (:issue:`6958`, :issue:`7586`)++- CI and test improvements and fixes.+ (:issue:`5954`,+ :issue:`7002`,+ :issue:`7017`,+ :issue:`7247`,+ :issue:`7508`,+ :issue:`7545`,+ :issue:`7566`,+ :issue:`7574`,+ :issue:`7585`,+ :issue:`7595`,+ :issue:`7608`,+ :issue:`7610`,+ :issue:`7612`,+ :issue:`7616`,+ :issue:`7625`,+ :issue:`7637`,+ :issue:`7639`,+ :issue:`7640`,+ :issue:`7641`,+ :issue:`7642`,+ :issue:`7643`,+ :issue:`7644`,+ :issue:`7645`,+ :issue:`7646`,+ :issue:`7654`,+ :issue:`7655`,+ :issue:`7664`,+ :issue:`7672`,+ :issue:`7677`,+ :issue:`7680`,+ :issue:`7682`,+ :issue:`7692`)+.. _release-2.16.0:Scrapy 2.16.0 (2026-05-19)@@ -6243,8 +6547,8 @@ Backward-incompatible changes``429``, you must override :setting:`RETRY_HTTP_CODES` accordingly.* :class:`~scrapy.crawler.Crawler`,- :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and- :class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`+ :meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and+ :meth:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, theyonly accept a :class:`~scrapy.spiders.Spider` subclass now.diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rstindex d0321c906..787ef842a 100644--- a/docs/topics/autothrottle.rst+++ b/docs/topics/autothrottle.rst@@ -88,6 +88,8 @@ server) is, and this extension builds on that premise.Prevent specific requests from triggering slot delay adjustments================================================================+.. versionadded:: 2.12.0+AutoThrottle adjusts the delay of download slots based on the latencies ofresponses that belong to that download slot. The only exceptions are non-200responses, which are only taken into account to increase that delay, butdiff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex c901e01e4..9bbfee9e2 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -329,13 +329,13 @@ Other limitations:HTTPX_HTTP2_ENABLED^^^^^^^^^^^^^^^^^^^+.. versionadded:: 2.17.0+Default: ``False``Whether to enable HTTP/2 support in this handler. The ``httpx[http2]`` extraneeds to be installed if you want to enable this setting.-.. versionadded:: VERSION-Built-in non-HTTP download handlers reference=============================================diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 5649453b1..054c1130c 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -351,6 +351,8 @@ HttpAuthMiddlewareHTTPAUTH_USER~~~~~~~~~~~~~+.. versionadded:: 2.17.0+Default: ``""``The username to use for HTTP basic authentication, applied to all requests@@ -361,6 +363,8 @@ whose URL matches :setting:`HTTPAUTH_DOMAIN`.HTTPAUTH_PASS~~~~~~~~~~~~~+.. versionadded:: 2.17.0+Default: ``""``The password to use for HTTP basic authentication.@@ -370,6 +374,8 @@ The password to use for HTTP basic authentication.HTTPAUTH_DOMAIN~~~~~~~~~~~~~~~+.. versionadded:: 2.17.0+Default: ``None``The domain (and its subdomains) to which HTTP basic authentication credentials@@ -1104,6 +1110,8 @@ exception propagation, seeRETRY_GIVE_UP_LOG_LEVEL^^^^^^^^^^^^^^^^^^^^^^^+.. versionadded:: 2.17.0+Default: ``"ERROR"``:ref:`Logging level <levels>` used for the message logged when a requestdiff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 6ce5255b9..9aa1b8d5f 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -528,10 +528,6 @@ safe numeric encoding (``\uXXXX`` sequences) for historic reasons.Use ``"utf-8"`` if you want UTF-8 for JSON too.-.. versionchanged:: 2.8- The :command:`startproject` command now sets this setting to- ``"utf-8"`` in the generated ``settings.py`` file.-.. setting:: FEED_EXPORT_FIELDSFEED_EXPORT_FIELDSdiff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex f2fcfb0b5..44bc09b7f 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -806,6 +806,8 @@ Whether or not to fail on broken responses. See:give_up_log_level-----------------+.. versionadded:: 2.17.0+:ref:`Logging level <levels>` used for the message logged when a requestexceeds its retries. See :setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.@@ -814,6 +816,8 @@ exceeds its retries. See :setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.http_auth_domain----------------+.. versionadded:: 2.17.0+Overrides :setting:`HTTPAUTH_DOMAIN` for this request... reqmeta:: http_pass@@ -821,6 +825,8 @@ Overrides :setting:`HTTPAUTH_DOMAIN` for this request.http_pass---------+.. versionadded:: 2.17.0+Overrides :setting:`HTTPAUTH_PASS` for this request... reqmeta:: http_user@@ -828,6 +834,8 @@ Overrides :setting:`HTTPAUTH_PASS` for this request.http_user---------+.. versionadded:: 2.17.0+Overrides :setting:`HTTPAUTH_USER` for this request... reqmeta:: max_retry_times@@ -844,6 +852,8 @@ The meta key is used set retry times per request. When set, theverbatim_url------------+.. versionadded:: 2.17.0+Set this key to ``True`` to keep the request URL as passed to:class:`~scrapy.Request`, without URL percent-encoding.@@ -854,7 +864,6 @@ characters that would otherwise be canonicalized get different fingerprints.In this mode, the ``keep_fragments`` parameter is ignored, and it iseffectively true.-.. _topics-stop-response-download:Stopping the download of a Responsediff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 11251a92c..a5148ee26 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -766,6 +766,9 @@ specific cipher that is not included in ``DEFAULT`` if a website requires it.Set this setting to ``None`` to use the default ciphers of the underlying TLSimplementation.+.. versionchanged:: 2.17.0+ Added support for setting this to ``None``.+.. _OpenSSL cipher list format: https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format.. note::@@ -781,6 +784,8 @@ implementation.DOWNLOAD_TLS_MAX_VERSION------------------------+.. versionadded:: 2.17.0+Default: ``None``Use this setting to change the maximum version of the TLS protocol allowed to@@ -816,6 +821,8 @@ modern environments.DOWNLOAD_TLS_MIN_VERSION------------------------+.. versionadded:: 2.17.0+Default: ``None``Use this setting to change the minimum version of the TLS protocol allowed todiff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.pyindex a0a0b60a2..e1dcd90d5 100644--- a/scrapy/downloadermiddlewares/retry.py+++ b/scrapy/downloadermiddlewares/retry.py@@ -87,6 +87,9 @@ def get_retry_request(message logged when a request exceeds its retries. See:setting:`RETRY_GIVE_UP_LOG_LEVEL` for details.+ .. versionadded:: 2.17.0+ The *give_up_log_level* parameter.+*stats_base_key* is a string to be used as the base key for theretry-related job stats"""diff --git a/scrapy/item.py b/scrapy/item.pyindex 4d99ea79d..57d2272e2 100644--- a/scrapy/item.py+++ b/scrapy/item.py@@ -100,7 +100,7 @@ class Item(MutableMapping[str, Any], object_ref, metaclass=ItemMeta):#: come first, followed by fields declared in subclasses, and a field#: redefined in a subclass keeps the position of its first definition.#:- #: .. versionchanged:: VERSION+ #: .. versionchanged:: 2.17.0#: Fields are now returned in definition order rather than alphabetical#: order.fields: dict[str, Field]diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.pyindex 2efe7cd8e..3fb741d7a 100644--- a/scrapy/linkextractors/lxmlhtml.py+++ b/scrapy/linkextractors/lxmlhtml.py@@ -235,7 +235,7 @@ class LxmlLinkExtractor:can be combined with ``tags='*'`` to consider every tag except a few.Defaults to ``()`` (no tag is excluded).- .. versionadded:: VERSION+ .. versionadded:: 2.17.0:type deny_tags: str or list:param deny_attrs: an attribute or a list of attributes that should not be@@ -244,7 +244,7 @@ class LxmlLinkExtractor:every attribute except a few. Defaults to ``()`` (no attribute isexcluded).- .. versionadded:: VERSION+ .. versionadded:: 2.17.0:type deny_attrs: str or list:param canonicalize: canonicalize each extracted url (using