Scrapy, a fast high-level web crawling & scraping framework for Python.
Release notes for Scrapy 2.16.0. (#7536)
Andrey Rakhmatullina84b7850fc11
diff --git a/docs/news.rst b/docs/news.rstindex 3173dfbc1..429ea09bf 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,8 +3,31 @@Release notes=============-Scrapy VERSION (unreleased)----------------------------+.. _release-2.16.0:++Scrapy 2.16.0 (unreleased)+--------------------------++Highlights:++- Official support for Python 3.14++- Support for Twisted 26.4.0+++Modified requirements+~~~~~~~~~~~~~~~~~~~~~++- Increased the minimum versions of the following dependencies:++ - service_identity_: 18.1.0 → 23.1.0++ (:issue:`7347`)++- Added support for Twisted 26.4.0+.+ (:issue:`7347`, :issue:`7505`, :issue:`7520`)++- Added support for Python 3.14.+ (:issue:`6604`, :issue:`7460`)Backward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~@@ -26,7 +49,237 @@ Backward-incompatible changes- ``scrapy.core.downloader.handlers.http2.ScrapyH2Agent``- (:issue:`7496`, #TBD)+ (:issue:`7496`, :issue:`7510`)++Deprecations+~~~~~~~~~~~~++- ``scrapy.FormRequest`` is deprecated. You can use the :doc:`form2request+ <form2request:index>` library instead, see :ref:`form`.+ (:issue:`6438`)++- ``scrapy.utils.python.MutableChain`` is deprecated.+ (:issue:`7504`)++Deprecation removals+~~~~~~~~~~~~~~~~~~~~++- The ``start_requests()`` method of :class:`~scrapy.Spider`, deprecated in+ 2.13.0, is removed and no longer called. Use :meth:`~scrapy.Spider.start`+ instead, or both to maintain support for lower Scrapy versions.+ (:issue:`7490`)++- Support for ``process_start_requests()`` methods of :ref:`spider middlewares+ <topics-spider-middleware>`, deprecated in 2.13.0, is removed. Use+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,+ or both to maintain support for lower Scrapy versions.+ (:issue:`7490`)++- Support for synchronous ``process_spider_output()`` methods of spider+ middlewares, deprecated in Scrapy 2.13.0, is removed. You should upgrade+ the affected middlewares to have asynchronous ``process_spider_output()``+ methods.+ (:issue:`7504`)++- The ``spider`` arguments of the following methods of+ :class:`~scrapy.core.scraper.Scraper`, deprecated in Scrapy 2.13.0, are+ removed:++ - ``close_spider()``++ - ``enqueue_scrape()``++ - ``handle_spider_error()``++ - ``handle_spider_output()``++ (:issue:`7487`)++- HTTP/1.0 support code, deprecated in Scrapy 2.13.0, is removed. This+ includes:++ - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``++ - The ``scrapy.core.downloader.webclient`` module.++ - The ``DOWNLOADER_HTTPCLIENTFACTORY`` setting.++ (:issue:`7486`)++- The following functions, deprecated in Scrapy 2.13.0, are removed, you+ should import them from :mod:`w3lib.url` directly instead:++ - ``scrapy.utils.url.add_or_replace_parameter()``++ - ``scrapy.utils.url.add_or_replace_parameters()``++ - ``scrapy.utils.url.any_to_uri()``++ - ``scrapy.utils.url.canonicalize_url()``++ - ``scrapy.utils.url.file_uri_to_path()``++ - ``scrapy.utils.url.is_url()``++ - ``scrapy.utils.url.parse_data_uri()``++ - ``scrapy.utils.url.parse_url()``++ - ``scrapy.utils.url.path_to_file_uri()``++ - ``scrapy.utils.url.safe_download_url()``++ - ``scrapy.utils.url.safe_url_string()``++ - ``scrapy.utils.url.url_query_cleaner()``++ - ``scrapy.utils.url.url_query_parameter()``++ (:issue:`7487`)++- The following test-related code, deprecated in Scrapy 2.13.0, is removed:++ - the ``scrapy.utils.testproc`` module++ - the ``scrapy.utils.testsite`` module++ - ``scrapy.utils.test.assert_gcs_environ()``++ - ``scrapy.utils.test.get_ftp_content_and_delete()``++ - ``scrapy.utils.test.get_gcs_content_and_delete()``++ - ``scrapy.utils.test.mock_google_cloud_storage()``++ - ``scrapy.utils.test.skip_if_no_boto()``++ - ``scrapy.utils.test.TestSpider``++ (:issue:`7487`)++- ``scrapy.utils.versions.scrapy_components_versions()``, deprecated in+ Scrapy 2.13.0, is removed, you can use+ :func:`scrapy.utils.versions.get_versions` instead.+ (:issue:`7487`)++- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` and+ ``scrapy.utils.url.escape_ajax()``, deprecated in Scrapy 2.13.0, are+ removed.+ (:issue:`7487`)++- The ``__init__()`` method of priority queue classes (see+ :setting:`SCHEDULER_PRIORITY_QUEUE`) now needs to support a keyword-only+ ``start_queue_cls`` parameter, not supporting it was deprecated in Scrapy+ 2.13.0.+ (:issue:`7487`)++- ``scrapy.spiders.init.InitSpider``, deprecated in Scrapy 2.13.0, is+ removed.+ (:issue:`7487`)++New features+~~~~~~~~~~~~++- New features and improvements for+ :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`:++ - Support for proxies.++ - Support for the :reqmeta:`download_latency` meta key.++ - Support for :attr:`Response.certificate+ <scrapy.http.Response.certificate>`.++ - Default headers set by the ``httpx`` library are no longer added to+ requests.++ (:issue:`7441`, :issue:`7524`)++- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now+ skips HTTPS proxy certificate verification when the+ :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting is set to ``False``.+ (:issue:`7496`)++Improvements+~~~~~~~~~~~~++- :func:`time.monotonic` is used instead of :func:`time.time` to calculate+ elapsed time in various places.+ (:issue:`7377`)++- Improved extraction of the file extension from the URL in+ :class:`~scrapy.pipelines.files.FilesPipeline`.+ (:issue:`4225`, :issue:`7414`)++- Other code refactoring and improvements.+ (:issue:`7401`)++Bug fixes+~~~~~~~~~++- :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now+ raises an exception when a request has an ``https://`` destination and an+ ``https://`` proxy, which is not supported by this handler. Previously it+ tried to connect to the proxy via HTTP in this case.+ (:issue:`7496`)++- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now+ raises an exception for requests with ``http://`` URLs instead of trying to+ connect, which is not supported by this handler.+ (:issue:`7496`)++- :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no longer+ adds the ``:status`` pseudo-header to :attr:`Response.headers+ <scrapy.http.Response.headers>`.+ (:issue:`7441`)++- Fixed :func:`scrapy.utils.response.open_in_browser` removing the ``<head>``+ tag when adding the ``<base>`` tag.+ (:issue:`7459`)++Documentation+~~~~~~~~~~~~~++- Documented that+ :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`+ doesn't support HTTPS proxies for HTTPS destinations and that+ :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` doesn't+ support proxies at all.+ (:issue:`7496`)++- Added an example of using+ :class:`logging.handlers.TimedRotatingFileHandler` to rotate Scrapy logs.+ (:issue:`3628`, :issue:`7501`)++- Added a ``CITATION.cff`` file.+ (:issue:`7502`, :issue:`7519`)++- Mentioned :setting:`DOWNLOADER_CLIENT_TLS_METHOD` in :ref:`bans`.+ (:issue:`5232`, :issue:`7518`)++- Other documentation improvements and fixes.+ (:issue:`7417`,+ :issue:`7463`,+ :issue:`7472`,+ :issue:`7480`,+ :issue:`7489`,+ :issue:`7503`,+ :issue:`7507`)++Quality assurance+~~~~~~~~~~~~~~~~~++- Added tests that connect to https://books.toscrape.com/ to test the+ behavior with a real website. These tests are marked with the+ ``requires_internet`` pytest mark and can be skipped with e.g.+ ``-m 'not requires_internet'`` if you cannot or don't want to run them.+ (:issue:`7520`)++- Type hints improvements and fixes.+ (:issue:`7492`, :issue:`7532`)++- CI and test improvements and fixes.+ (:issue:`7441`, :issue:`7466`, :issue:`7491`, :issue:`7496`).. _release-2.15.2:diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 820d5910c..99bbdf292 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -175,9 +175,9 @@ In Scrapy 2.6.3 and lower, ``process_spider_output()`` must be a *synchronous*generator.To support those versions and higher Scrapy versions in the same middleware,-rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()`-method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define-a synchronous ``process_spider_output()`` method to be used by 2.6.3 and lower+rename your asynchronous :meth:`~SpiderMiddleware.process_spider_output`+method to :meth:`~SpiderMiddleware.process_spider_output_async`, and define a+synchronous ``process_spider_output()`` method to be used by 2.6.3 and lowerversions.For example: