Scrapy, a fast high-level web crawling & scraping framework for Python.
Release notes for 2.14.0 (#7035)
Andrey Rakhmatullin7bef98b4f14a
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yamlindex 5ffe2a8ea..dc4276cff 100644--- a/.pre-commit-config.yaml+++ b/.pre-commit-config.yaml@@ -23,6 +23,6 @@ repos:- id: end-of-file-fixer- id: trailing-whitespace- repo: https://github.com/sphinx-contrib/sphinx-lint- rev: v1.0.0+ rev: v1.0.2hooks:- id: sphinx-lintdiff --git a/docs/news.rst b/docs/news.rstindex 7e15c2167..1b1c72c1b 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,8 +3,41 @@Release notes=============-Scrapy VERSION (unreleased)----------------------------+.. _release-2.14.0:++Scrapy 2.14.0 (unreleased)+--------------------------++Highlights:++- More coroutine-based replacements for Deferred-based APIs++- The default priority queue is now ``DownloaderAwarePriorityQueue``++- Dropped support for Python 3.9 and PyPy 3.10++- Improved and documented the API for custom download handlers++Modified requirements+~~~~~~~~~~~~~~~~~~~~~++- Dropped support for Python 3.9.+ (:issue:`7121`)++- Dropped support for PyPy 3.10.+ (:issue:`7050`)++- Increased the minimum versions of the following dependencies:++ - lxml_: 4.6.0 → 4.6.4++ - Pillow_ (optional dependency): 8.0.0 → 8.3.2++ - botocore_ (optional dependency): 1.4.87 → 1.13.45++- Restored support for ``brotlicffi`` dropped in Scrapy 2.13.4. Its minimum+ supported version is now ``1.2.0.0``.+ (:issue:`7160`)Backward-incompatible changes~~~~~~~~~~~~~~~~~~~~~~~~~~~~~@@ -14,6 +47,7 @@ Backward-incompatible changesmay now need to set the :setting:`FORCE_CRAWLER_PROCESS` setting to``True`` when running Scrapy via :ref:`its command-line tool<topics-commands-crawlerprocess>` to avoid a reactor mismatch exception.+ (:issue:`6845`)- The ``log_count/*`` stats no longer count some of the early messages thatthey counted before. While the earliest log messages, emitted before the@@ -52,6 +86,17 @@ Backward-incompatible changes- :meth:`~scrapy.spidermiddlewares.referer.ReferrerPolicy.referrer`+ (:issue:`6930`)++- Scrapy no longer passes a ``spider`` argument to any methods of the+ :setting:`stats collector <STATS_CLASS>`. It wasn't passed in many of the+ calls even in older Scrapy versions, so we don't expect existing custom+ stats collector implementations to require a ``spider`` argument. If your+ implementation needs a :class:`~scrapy.Spider` instance, you can get it+ from the :class:`~scrapy.crawler.Crawler` instance passed to the+ constructor.+ (:issue:`7011`)+- :class:`scrapy.middleware.MiddlewareManager` no longer includes code forhandling ``open_spider()`` and ``close_spider()`` component methods. Asthis code was only used for pipelines it was moved into@@ -66,6 +111,8 @@ Backward-incompatible changes- Code in ``scrapy.middleware.MiddlewareManager._add_middleware()`` thatprocesses ``open_spider()`` and ``close_spider()`` component methods.+ (:issue:`7006`)+- :meth:`scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware.process_request`now returns a coroutine, previously it returned a:class:`~twisted.internet.defer.Deferred` object or ``None``. The@@ -73,10 +120,11 @@ Backward-incompatible changeschange only impacts code that subclasses:class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` orcalls its methods directly.+ (:issue:`6802`)-- The built-in :ref:`download handlers <download-handlers-ref>` were- refactored, changing signatures of their methods. This change should only- affect user code that subclasses any of these handlers or calls their+- The built-in :ref:`download handlers <download-handlers-ref>` have been+ refactored, changing the signatures of their methods. This change should+ only affect user code that subclasses any of these handlers or calls theirmethods directly.(:issue:`6778`, :issue:`7164`)@@ -84,6 +132,549 @@ Backward-incompatible changescoroutine, previously it returned a:class:`~twisted.internet.defer.Deferred` object. Thischange only impacts code that calls this method directly.+ (:issue:`7177`)++Deprecation removals+~~~~~~~~~~~~~~~~~~~~++- The ``from_settings()`` method of the following components, deprecated in+ Scrapy 2.12.0, is removed. You should use ``from_crawler()`` instead.++ - :class:`scrapy.dupefilters.RFPDupeFilter`+ - :class:`scrapy.mail.MailSender`+ - :class:`scrapy.middleware.MiddlewareManager`+ - :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`+ - :class:`scrapy.pipelines.files.FilesPipeline`+ - :class:`scrapy.pipelines.images.ImagesPipeline`++ (:issue:`7126`)++- Scrapy no longer calls ``from_settings()`` methods of 3rd-party+ :ref:`components <topics-components>`, deprecated in Scrapy 2.12.0. You+ should define a ``from_crawler()`` method instead.+ (:issue:`7126`)++- The initialization flow of :class:`scrapy.pipelines.media.MediaPipeline`+ and its subclasses was simplified, it now mandates ``from_crawler()``+ methods and ``crawler`` arguments of ``__init__()`` methods. Not using+ these was deprecated in Scrapy 2.12.0.+ (:issue:`7126`)++- The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting, deprecated in Scrapy+ 2.12.0, is removed.+ (:issue:`7126`)++- The ``scrapy.utils.misc.create_instance()`` function, deprecated in Scrapy+ 2.12.0, is removed. Use :func:`scrapy.utils.misc.build_from_crawler`+ instead.+ (:issue:`7126`)++- The ``scrapy.core.downloader.Downloader._get_slot_key()`` function,+ deprecated in Scrapy 2.12.0, is removed. Use+ :meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.+ (:issue:`7126`)++- The ``scrapy.twisted_version`` attribute, deprecated in Scrapy 2.12.0, is+ removed. You should instead use the :attr:`twisted.version` attribute+ directly.+ (:issue:`7126`)++- The following utility functions, deprecated in Scrapy 2.12.0, are removed:++ - ``scrapy.utils.defer.process_chain_both()``+ - ``scrapy.utils.python.equal_attributes()``+ - ``scrapy.utils.python.flatten()``+ - ``scrapy.utils.python.iflatten()``+ - ``scrapy.utils.request.request_authenticate()``+ - ``scrapy.utils.test.assert_samelines()``++ (:issue:`7126`)++- ``scrapy.utils.serialize.ScrapyJSONDecoder``, deprecated in Scrapy 2.12.0,+ is removed.+ (:issue:`7126`)++- The ``scrapy.extensions.feedexport.build_storage()`` function, deprecated+ in Scrapy 2.12.0, is removed, you can instead call the builder callable+ directly.+ (:issue:`7126`)++- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware``, deprecated in+ Scrapy 2.11.2, is removed.+ :class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` should be+ used instead.+ (:issue:`6926`)++Deprecations+~~~~~~~~~~~~++- The following methods that return a+ :class:`~twisted.internet.defer.Deferred` are deprecated in favor of their+ coroutine-based replacements:++ - :class:`scrapy.core.downloader.handlers.DownloadHandlers`++ - ``download_request()`` (use+ :meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`)++ - :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`++ - ``download()`` (use+ :meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`)++ - :class:`scrapy.core.engine.ExecutionEngine`++ - ``start()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.start_async`)++ - ``stop()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.stop_async`)++ - ``close()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.close_async`)++ - ``open_spider()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async`)++ - ``close_spider()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async`)++ - ``download()`` (use+ :meth:`~scrapy.core.engine.ExecutionEngine.download_async`)++ - :class:`scrapy.core.scraper.Scraper`++ - ``open_spider()`` (use+ :meth:`~scrapy.core.scraper.Scraper.open_spider_async`)++ - ``call_spider()`` (use+ :meth:`~scrapy.core.scraper.Scraper.call_spider_async`)++ - ``close_spider()`` (use+ :meth:`~scrapy.core.scraper.Scraper.close_spider_async`)++ - ``handle_spider_output()`` (use+ :meth:`~scrapy.core.scraper.Scraper.handle_spider_output_async`)++ - ``start_itemproc()`` (use+ :meth:`~scrapy.core.scraper.Scraper.start_itemproc_async`)++ - :class:`scrapy.core.spidermw.SpiderMiddlewareManager`++ - ``scrape_response()`` (use+ :meth:`~scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response_async`)++ - :class:`scrapy.crawler.Crawler`++ - ``stop()`` (use :meth:`~scrapy.crawler.Crawler.stop_async`)++ - :class:`scrapy.pipelines.ItemPipelineManager`++ - ``process_item()`` (use+ :meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async`)++ - ``open_spider()`` (use+ :meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async`)++ - ``close_spider()`` (use+ :meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async`)++ - :class:`scrapy.signalmanager.SignalManager`++ - ``send_catch_log_deferred()`` (use+ :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async`)++ - ``scrapy.utils.signal.send_catch_log_deferred()`` (use+ :func:`scrapy.utils.signal.send_catch_log_async`)++ (:issue:`6791`, :issue:`6842`, :issue:`6979`, :issue:`6997`, :issue:`6999`,+ :issue:`7005`, :issue:`7043`, :issue:`7069`, :issue:`7161`, :issue:`7164`)++- The following spider attributes are deprecated in favor of settings:++ - ``download_maxsize`` (use :setting:`DOWNLOAD_MAXSIZE`)++ - ``download_timeout`` (use :setting:`DOWNLOAD_TIMEOUT`)++ - ``download_warnsize`` (use :setting:`DOWNLOAD_WARNSIZE`)++ - ``max_concurrent_requests`` (use :setting:`CONCURRENT_REQUESTS`)++ - ``user_agent`` (use :setting:`USER_AGENT`)++ (:issue:`6988`, :issue:`6994`, :issue:`7038`, :issue:`7039`, :issue:`7117`,+ :issue:`7176`)++- Returning a :class:`~twisted.internet.defer.Deferred` from the following+ user-defined functions is deprecated in favor of defining them as coroutine+ functions:++ - spider callbacks and errbacks (which was never officially supported and+ may work incorrectly)++ - the ``process_request()``, ``process_response()`` and+ ``process_exception()`` methods of custom downloader middlewares++ - the ``process_item()``, ``open_spider()`` and ``close_spider()`` methods+ of custom pipelines++ - signal handlers++ - the ``download_request()`` and ``close()`` methods of custom download+ handlers++ (:issue:`6718`, :issue:`6778`, :issue:`7069`, :issue:`7147`, :issue:`7148`,+ :issue:`7149`, :issue:`7150`, :issue:`7151`, :issue:`7161`, :issue:`7164`,+ :issue:`7179`)++- Passing a ``spider`` argument to the following methods is deprecated:++ - :meth:`scrapy.core.spidermw.SpiderMiddlewareManager.process_start`++ - :meth:`scrapy.core.downloader.Downloader.fetch`++ - :meth:`scrapy.core.downloader.Downloader._get_slot`++ - :meth:`scrapy.core.downloader.handlers.DownloadHandlers.download_request`++ - all public methods of :class:`scrapy.statscollectors.StatsCollector`++ - :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output`++ - :meth:`scrapy.spidermiddlewares.base.BaseSpiderMiddleware.process_spider_output_async`++ - all ``process_*()`` methods of built-in downloader middlewares++ - all ``process_*()`` methods of built-in spider middlewares++ - :meth:`scrapy.pipelines.media.MediaPipeline.open_spider`++ - :meth:`scrapy.pipelines.media.MediaPipeline.process_item`++ (:issue:`6750`, :issue:`6927`, :issue:`6984`, :issue:`7006`, :issue:`7011`,+ :issue:`7033`, :issue:`7037`, :issue:`7045`, :issue:`7178`)++- Instantiating subclasses of :class:`scrapy.middleware.MiddlewareManager`+ without a :class:`~scrapy.crawler.Crawler` instance is deprecated.+ (:issue:`6984`)++- For the following user-defined functions and methods requiring a ``spider``+ argument is deprecated, if you need a :class:`~scrapy.Spider` instance+ inside them you should get it from the :class:`~scrapy.crawler.Crawler`+ instance (you may need to refactor your code to save that instance in e.g.+ the ``from_crawler()`` method):++ - the ``process_request()``, ``process_response()`` and+ ``process_exception()`` methods of custom downloader middlewares++ - the ``process_spider_input()``, ``process_spider_output()``,+ ``process_spider_output_async()`` and ``process_spider_exception()``+ methods of custom spider middlewares++ - the ``process_item()`` method of custom pipelines++ - the ``fetch()`` method of a custom :setting:`DOWNLOADER`++ (:issue:`6927`, :issue:`6984`, :issue:`7006`, :issue:`7037`)++- The following things in custom download handlers are deprecated:++ - not having a ``lazy`` attribute (you should define it as ``True`` if you+ want to keep the current behavior)++ - returning a :class:`~twisted.internet.defer.Deferred` from the+ ``download_request()`` method (you should refactor it to return a+ coroutine; you also need to remove the ``spider`` argument when doing+ this)++ - not having a ``close()`` method, having a synchronous one or one that+ returns a :class:`~twisted.internet.defer.Deferred` (you should refactor+ it to return a coroutine or add an empty one if you don't have it)++ (:issue:`6778`, :issue:`7164`)++- Custom implementations of :setting:`ITEM_PROCESSOR` should now define+ ``process_item_async()``, ``open_spider_async()`` and+ ``close_spider_async()`` methods instead of, or in addition to,+ ``process_item()``, ``open_spider()`` and ``close_spider()``.+ (:issue:`7005`, :issue:`7043`)++- The ``CONCURRENT_REQUESTS_PER_IP`` setting is deprecated, use+ :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` instead.+ (:issue:`6917`, :issue:`6921`)++- The ``scrapy.core.downloader.handlers.http`` module is deprecated. You+ should import+ :class:`scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`+ directly instead of importing the+ ``scrapy.core.downloader.handlers.http.HTTPDownloadHandler`` alias.+ (:issue:`7079`)++- The ``scrapy.utils.decorators.defers()`` decorator is deprecated, you can+ use :func:`twisted.internet.defer.maybeDeferred` directly or reimplement+ this decorator in your code.+ (:issue:`7164`)++- ``scrapy.spiders.CrawlSpider._parse_response()`` is deprecated, use+ :meth:`scrapy.spiders.CrawlSpider.parse_with_rules` instead.+ (:issue:`4463`, :issue:`6804`)++- The functions that add a delay to a Deferred are deprecated, their+ underlying Twisted functions can be used instead, either directly if a+ delay isn't needed, or with some explicit way to add a delay if it's+ needed:++ - ``scrapy.utils.defer.mustbe_deferred()`` (you can use+ :func:`twisted.internet.defer.maybeDeferred`)++ - ``scrapy.utils.defer.defer_succeed()`` (you can use+ :func:`twisted.internet.defer.succeed`)++ - ``scrapy.utils.defer.defer_fail()`` (you can use+ :func:`twisted.internet.defer.fail`)++ - ``scrapy.utils.defer.defer_result()`` (you can use+ :func:`twisted.internet.defer.succeed` and+ :func:`twisted.internet.defer.fail`)++ (:issue:`6937`)++New features+~~~~~~~~~~~~++- Added :class:`scrapy.crawler.AsyncCrawlerProcess` and+ :class:`scrapy.crawler.AsyncCrawlerRunner` as counterparts to+ :class:`~scrapy.crawler.CrawlerProcess` and+ :class:`~scrapy.crawler.CrawlerRunner` that offer coroutine-based APIs.+ (:issue:`6789`, :issue:`6790`, :issue:`6796`, :issue:`6817`, :issue:`6845`,+ :issue:`7034`)++- Added coroutine counterparts to some of the Deferred-based APIs:++ - :class:`scrapy.core.downloader.handlers.DownloadHandlers`++ - :meth:`~scrapy.core.downloader.handlers.DownloadHandlers.download_request_async`+ (to ``download_request()``)++ - :class:`scrapy.core.downloader.middleware.DownloaderMiddlewareManager`++ - :meth:`~scrapy.core.downloader.middleware.DownloaderMiddlewareManager.download_async`+ (to ``download()``)++ - :class:`scrapy.core.engine.ExecutionEngine`++ - :meth:`~scrapy.core.engine.ExecutionEngine.start_async` (to+ ``start()``)++ - :meth:`~scrapy.core.engine.ExecutionEngine.stop_async` (to+ ``stop()``)++ - :meth:`~scrapy.core.engine.ExecutionEngine.close_async` (to+ ``close()``)++ - :meth:`~scrapy.core.engine.ExecutionEngine.open_spider_async` (to+ ``open_spider()``)++ - :meth:`~scrapy.core.engine.ExecutionEngine.close_spider_async` (to+ ``close_spider()``)++ - :meth:`~scrapy.core.engine.ExecutionEngine.download_async` (to+ ``download()``)++ - :class:`scrapy.core.scraper.Scraper`++ - :meth:`~scrapy.core.scraper.Scraper.open_spider_async` (to+ ``open_spider()``)++ - :meth:`~scrapy.core.scraper.Scraper.close_spider_async` (to+ ``close_spider()``)++ - :meth:`~scrapy.core.scraper.Scraper.start_itemproc_async` (to+ ``start_itemproc()``)++ - :class:`scrapy.crawler.Crawler`++ - :meth:`~scrapy.crawler.Crawler.crawl_async` (to ``crawl()``)++ - :meth:`~scrapy.crawler.Crawler.stop_async` (to ``stop()``)++ - :class:`scrapy.pipelines.ItemPipelineManager`++ - :meth:`~scrapy.pipelines.ItemPipelineManager.process_item_async` (to+ ``process_item()``)++ - :meth:`~scrapy.pipelines.ItemPipelineManager.open_spider_async` (to+ ``open_spider()``)++ - :meth:`~scrapy.pipelines.ItemPipelineManager.close_spider_async` (to+ ``close_spider()``)++ - :class:`scrapy.signalmanager.SignalManager`++ - :meth:`~scrapy.signalmanager.SignalManager.send_catch_log_async` (to+ ``send_catch_log_deferred()``)++ (:issue:`6781`, :issue:`6791`, :issue:`6792`, :issue:`6795`, :issue:`6801`,+ :issue:`6817`, :issue:`6842`, :issue:`6997`, :issue:`7005`, :issue:`7043`,+ :issue:`7069`,:issue:`7164`, :issue:`7202`)++- The default value of the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is now+ ``'scrapy.pqueues.DownloaderAwarePriorityQueue'``.+ (:issue:`6924`, :issue:`6940`)++- Added :class:`scrapy.extensions.logcount.LogCount`, an enabled-by-default+ extension that is responsible for the ``log_count/*`` stats. Previously,+ this code was in :class:`scrapy.crawler.Crawler` and couldn't be disabled.+ (:issue:`7046`)++- Added :meth:`scrapy.spiders.CrawlSpider.parse_with_rules` as a public+ replacement for ``_parse_response()``.+ (:issue:`4463`, :issue:`6804`)++- Added :func:`scrapy.utils.asyncio.is_asyncio_available` as an alternative+ to :func:`scrapy.utils.defer.is_asyncio_reactor_installed` with a+ future-proof name and semantics.+ (:issue:`6827`)++- The API for :ref:`download handlers <topics-download-handlers>`, previously+ undocumented, has been modernized and documented. An optional base class,+ :class:`scrapy.core.downloader.handlers.base.BaseDownloadHandler`, has been+ added to simplify writing custom download handlers that conform to the+ current API.+ (:issue:`4944`, :issue:`6778`, :issue:`7164`)++- Added :func:`scrapy.utils.defer.ensure_awaitable`, which can be helpful to+ call user-defined functions that can return coroutines, Deferreds or+ values directly.+ (:issue:`7005`)++- The ``requests.seen`` file, written by+ :class:`~scrapy.dupefilters.RFPDupeFilter` when :ref:`job persistence+ <topics-jobs>` is enabled, now uses line buffering to reduce data loss in+ spider crashes.+ (:issue:`6019`, :issue:`7094`)++- Images downloaded by :class:`~scrapy.pipelines.images.ImagesPipeline` are+ now automatically transposed based on EXIF data.+ (:issue:`6525`, :issue:`6975`)++Improvements+~~~~~~~~~~~~++- Refactored internal functions to use coroutines instead of Deferreds.+ (:issue:`6795`, :issue:`6852`, :issue:`6855`, :issue:`6858`, :issue:`7159`)++- Commands that don't need a :class:`~scrapy.crawler.CrawlerProcess` instance+ no longer create it.+ (:issue:`6824`)++- Improved :command:`shell` help formatting when using IPython 9+.+ (:issue:`6915`, :issue:`6980`)++Bug fixes+~~~~~~~~~++- Setting :setting:`FILES_STORE` or :setting:`IMAGES_STORE` to ``None`` now+ correctly disables the respective pipeline.+ (:issue:`6964`, :issue:`6969`)++- :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` now+ uses the URL set in the ``<base>`` tag as the base URL when redirecting to+ a relative URL.+ (:issue:`7042`, :issue:`7047`)++- Passing ``None`` as a value of the :reqmeta:`download_slot` request meta+ key is now handled in the same way as not setting this meta key at all.+ (:issue:`7172`)++- Fixed parsing of the first line of ``robots.txt`` files that have a BOM.+ (:issue:`6195`, :issue:`7095`)++Documentation+~~~~~~~~~~~~~++- Added :ref:`documentation <topics-download-handlers>` about download+ handlers, their API and built-in handlers.+ (:issue:`4944`, :issue:`7164`)++- Added a section about the `scrapy-spider-metadata`_ library to the+ :ref:`spider argument docs <spiderargs-scrapy-spider-metadata>`.+ (:issue:`6676`, :issue:`6957`, :issue:`7116`)++ .. _scrapy-spider-metadata: https://scrapy-spider-metadata.readthedocs.io/en/latest/++- Improved :ref:`the docs <coroutine-deferred-apis>` about coroutine-based+ and Deferred-based APIs.+ (:issue:`6800`, :issue:`7146`)++- Other documentation improvements and fixes.+ (:issue:`7058`, :issue:`7076`, :issue:`7109`, :issue:`7195`, :issue:`7198`)++Quality assurance+~~~~~~~~~~~~~~~~~++- Switched from ``twisted.trial`` to ``pytest-twisted`` and replaced+ remaining ``unittest`` and ``twisted.trial`` features with ``pytest`` ones.+ (:issue:`6658`, :issue:`6873`, :issue:`6884`, :issue:`6938`)++- Enabled fancy ``pytest`` asserts.+ (:issue:`6888`)++- Added `Sphinx Lint`_ to the ``pre-commit`` configuration.+ (:issue:`6920`)++ .. _Sphinx Lint: https://github.com/sphinx-contrib/sphinx-lint++- CI and test improvements and fixes.+ (:issue:`6649`,+ :issue:`6769`,+ :issue:`6821`,+ :issue:`6835`,+ :issue:`6836`,+ :issue:`6846`,+ :issue:`6883`,+ :issue:`6885`,+ :issue:`6889`,+ :issue:`6905`,+ :issue:`6928`,+ :issue:`6933`,+ :issue:`6941`,+ :issue:`6942`,+ :issue:`6945`,+ :issue:`6947`,+ :issue:`6960`,+ :issue:`6968`,+ :issue:`6972`,+ :issue:`6974`,+ :issue:`6996`,+ :issue:`7003`,+ :issue:`7012`,+ :issue:`7013`,+ :issue:`7050`,+ :issue:`7059`,+ :issue:`7070`,+ :issue:`7073`,+ :issue:`7118`,+ :issue:`7127`,+ :issue:`7141`,+ :issue:`7143`,+ :issue:`7145`,+ :issue:`7173`)++- Code cleanups.+ (:issue:`6803`,+ :issue:`6838`,+ :issue:`6849`,+ :issue:`6875`,+ :issue:`6876`,+ :issue:`6892`,+ :issue:`6930`,+ :issue:`6949`,+ :issue:`6970`,+ :issue:`6977`,+ :issue:`6986`,+ :issue:`7008`,+ :issue:`7177`).. _release-2.13.4:@@ -265,7 +856,7 @@ Highlights:- The asyncio reactor is now enabled by default- Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`- (async) and changed how it is iterated.+ (async) and changed how it is iterated- Added the :reqmeta:`allow_offsite` request meta key@@ -466,9 +1057,9 @@ Deprecations- The following modules and functions used only in tests are deprecated:- - the ``scrapy/utils/testproc`` module+ - the ``scrapy.utils.testproc`` module- - the ``scrapy/utils/testsite`` module+ - the ``scrapy.utils.testsite`` module- ``scrapy.utils.test.assert_gcs_environ()``@@ -6969,7 +7560,7 @@ Enhancements- Make ``RFPDupeFilter`` class easily subclassable (:issue:`533`)- Improve test coverage and forthcoming Python 3 support (:issue:`525`)- Promote startup info on settings and middleware to INFO level (:issue:`520`)-- Support partials in ``get_func_args`` util (:issue:`506`, issue:`504`)+- Support partials in ``get_func_args`` util (:issue:`506`, :issue:`504`)- Allow running individual tests via tox (:issue:`503`)- Update extensions ignored by link extractors (:issue:`498`)- Add middleware methods to get files/images/thumbs paths (:issue:`490`)@@ -7537,7 +8128,7 @@ API changes- ``scrapy.core.manager.ScrapyManager`` class renamed to ``scrapy.crawler.Crawler``- ``scrapy.core.manager.scrapymanager`` singleton moved to ``scrapy.project.crawler``- Moved module: ``scrapy.contrib.spidermanager`` to ``scrapy.spidermanager``-- Spider Manager singleton moved from ``scrapy.spider.spiders`` to the ``spiders` attribute of ``scrapy.project.crawler`` singleton.+- Spider Manager singleton moved from ``scrapy.spider.spiders`` to the ``spiders`` attribute of ``scrapy.project.crawler`` singleton.- moved Stats Collector classes: (#204)- ``scrapy.stats.collector.StatsCollector`` to ``scrapy.statscol.StatsCollector``- ``scrapy.stats.collector.SimpledbStatsCollector`` to ``scrapy.contrib.statscol.SimpledbStatsCollector``diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rstindex 36f9a1a53..b2b17a408 100644--- a/docs/topics/asyncio.rst+++ b/docs/topics/asyncio.rst@@ -97,6 +97,7 @@ Scrapy API requires passing a Deferred to it) using the following helpers:.. autofunction:: scrapy.utils.defer.deferred_from_coro.. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f+.. autofunction:: scrapy.utils.defer.ensure_awaitable.. _enforce-asyncio-requirement:diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rstindex a78e98828..a2dd67f68 100644--- a/docs/topics/coroutines.rst+++ b/docs/topics/coroutines.rst@@ -57,7 +57,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):- Methods of :ref:`download handlers <topics-download-handlers>`.- .. versionadded:: VERSION+ .. versionadded:: 2.14.. _coroutine-deferred-apis:diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rstindex 5f00d7131..7e672f0b4 100644--- a/docs/topics/download-handlers.rst+++ b/docs/topics/download-handlers.rst@@ -72,7 +72,7 @@ Writing your own download handler=================================A download handler is a :ref:`component <topics-components>` that defines-the following methods:+the following API:.. class:: SampleDownloadHandlerdiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 699b141c4..98374b6b4 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -362,6 +362,8 @@ used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`::Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.See `Scrapyd documentation`_.+.. _spiderargs-scrapy-spider-metadata:+scrapy-spider-metadata parameters---------------------------------diff --git a/scrapy/core/engine.py b/scrapy/core/engine.pyindex ffaddc2c4..4f8d8e8a1 100644--- a/scrapy/core/engine.py+++ b/scrapy/core/engine.py@@ -172,7 +172,7 @@ class ExecutionEngine:async def start_async(self, *, _start_request_processing: bool = True) -> None:"""Start the execution engine.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if self._starting:raise RuntimeError("Engine already running")@@ -208,7 +208,7 @@ class ExecutionEngine:async def stop_async(self) -> None:"""Gracefully stop the execution engine.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if not self._starting:@@ -459,7 +459,7 @@ class ExecutionEngine:Only downloader middlewares are applied.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if self.spider is None:raise RuntimeError(f"No open spider to crawl: {request}")@@ -575,7 +575,7 @@ class ExecutionEngine:async def close_spider_async(self, *, reason: str = "cancelled") -> None:"""Close (cancel) spider and clear all its outstanding requests.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if self.spider is None:raise RuntimeError("Spider not opened")diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.pyindex 455326a0a..df6aee715 100644--- a/scrapy/core/scraper.py+++ b/scrapy/core/scraper.py@@ -162,7 +162,7 @@ class Scraper:async def open_spider_async(self) -> None:"""Open the spider for scraping and allocate resources for it.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""self.slot = Slot(self.crawler.settings.getint("SCRAPER_SLOT_MAX_ACTIVE_SIZE"))if not self.crawler.spider:@@ -187,7 +187,7 @@ class Scraper:async def close_spider_async(self) -> None:"""Close the spider being scraped and release its resources.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if self.slot is None:raise RuntimeError("Scraper slot not assigned")@@ -488,7 +488,7 @@ class Scraper:*response* is the source of the item data. If the item does not comefrom response data, e.g. it was hard-coded, set it to ``None``.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""assert self.slot is not None # typingassert self.crawler.spider is not None # typingdiff --git a/scrapy/crawler.py b/scrapy/crawler.pyindex e21207222..66148904b 100644--- a/scrapy/crawler.py+++ b/scrapy/crawler.py@@ -165,7 +165,7 @@ class Crawler:*args* and *kwargs* arguments, while setting the execution engine inmotion. Should be called only once.- .. versionadded:: VERSION+ .. versionadded:: 2.14Complete when the crawl is finished."""@@ -209,7 +209,7 @@ class Crawler:async def stop_async(self) -> None:"""Start a graceful stop of the crawler and complete when the crawler is stopped.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if self.crawling:self.crawling = Falsediff --git a/scrapy/extensions/logcount.py b/scrapy/extensions/logcount.pyindex 04e570bbf..e6d51a7d8 100644--- a/scrapy/extensions/logcount.py+++ b/scrapy/extensions/logcount.py@@ -22,7 +22,7 @@ class LogCount:The handler installed is :class:`scrapy.utils.log.LogCounterHandler`.The counts are stored in stats as ``log_count/<level>``.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""def __init__(self, crawler: Crawler):diff --git a/scrapy/http/response/__init__.py b/scrapy/http/response/__init__.pyindex de2188ceb..2cc35fef4 100644--- a/scrapy/http/response/__init__.py+++ b/scrapy/http/response/__init__.py@@ -205,9 +205,6 @@ class Response(object_ref)::class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`method which supports selectors in addition to absolute/relative URLsand Link objects.-- .. versionadded:: 2.0- The *flags* parameter."""if encoding is None:raise ValueError("encoding can't be None")@@ -250,8 +247,6 @@ class Response(object_ref):flags: list[str] | None = None,) -> Iterable[Request]:"""- .. versionadded:: 2.0-Return an iterable of :class:`~.Request` instances to follow all linksin ``urls``. It accepts the same arguments as ``Request.__init__()`` method,but elements of ``urls`` can be relative URLs or :class:`~scrapy.link.Link` objects,diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.pyindex 9c4e4c151..012ead519 100644--- a/scrapy/http/response/text.py+++ b/scrapy/http/response/text.py@@ -77,11 +77,7 @@ class TextResponse(Response):)def json(self) -> Any:- """- .. versionadded:: 2.2-- Deserialize a JSON document to a Python object.- """+ """Deserialize a JSON document to a Python object."""if self._cached_decoded_json is _NONE:self._cached_decoded_json = json.loads(self.body)return self._cached_decoded_jsondiff --git a/scrapy/logformatter.py b/scrapy/logformatter.pyindex e81a9ec93..a50064e08 100644--- a/scrapy/logformatter.py+++ b/scrapy/logformatter.py@@ -142,8 +142,6 @@ class LogFormatter:) -> LogFormatterResult:"""Logs a message when an item causes an error while it is passingthrough the item pipeline.-- .. versionadded:: 2.0"""return {"level": logging.ERROR,@@ -160,10 +158,7 @@ class LogFormatter:response: Response | Failure,spider: Spider,) -> LogFormatterResult:- """Logs an error message from a spider.-- .. versionadded:: 2.0- """+ """Logs an error message from a spider."""return {"level": logging.ERROR,"msg": SPIDERERRORMSG,@@ -182,8 +177,6 @@ class LogFormatter:) -> LogFormatterResult:"""Logs a download error message from a spider (typically coming fromthe engine).-- .. versionadded:: 2.0"""args: dict[str, Any] = {"request": request}if errmsg:diff --git a/scrapy/signalmanager.py b/scrapy/signalmanager.pyindex 30afa9168..3ca9718d7 100644--- a/scrapy/signalmanager.py+++ b/scrapy/signalmanager.py@@ -85,7 +85,7 @@ class SignalManager:The keyword arguments are passed to the signal handlers (connectedthrough the :meth:`connect` method).- .. versionadded:: VERSION+ .. versionadded:: 2.14"""# note that this returns exceptions instead of Failures in the second tuple memberkwargs.setdefault("sender", self.sender)diff --git a/scrapy/utils/asyncio.py b/scrapy/utils/asyncio.pyindex 25237d0ba..a8216bc2b 100644--- a/scrapy/utils/asyncio.py+++ b/scrapy/utils/asyncio.py@@ -33,7 +33,7 @@ logger = logging.getLogger(__name__)def is_asyncio_available() -> bool:"""Check if it's possible to call asyncio code that relies on the asyncio event loop.- .. versionadded:: VERSION+ .. versionadded:: 2.14Currently this function is identical to:func:`scrapy.utils.reactor.is_asyncio_reactor_installed`: it returnsdiff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.pyindex afbe680ef..af7c50d5c 100644--- a/scrapy/utils/defer.py+++ b/scrapy/utils/defer.py@@ -461,9 +461,7 @@ def deferred_to_future(d: Deferred[_T]) -> Future[_T]:deferred = self.crawler.engine.download(additional_request)additional_response = await deferred_to_future(deferred)- .. versionadded:: 2.6.0-- .. versionchanged:: VERSION+ .. versionchanged:: 2.14This function no longer installs an asyncio loop if called before theTwisted asyncio reactor is installed. A :exc:`RuntimeError` is raisedin this case.@@ -495,8 +493,6 @@ def maybe_deferred_to_future(d: Deferred[_T]) -> Deferred[_T] | Future[_T]:additional_request = scrapy.Request('https://example.org/price')deferred = self.crawler.engine.download(additional_request)additional_response = await maybe_deferred_to_future(deferred)-- .. versionadded:: 2.6.0"""if not is_asyncio_available():return d@@ -534,7 +530,7 @@ def ensure_awaitable(o: _T | Awaitable[_T], _warn: str | None = None) -> Awaitabawaitable object of a different type, return it as is. For any othervalue, return a coroutine that completes with that value.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if isinstance(o, Deferred):if _warn:diff --git a/scrapy/utils/deprecate.py b/scrapy/utils/deprecate.pyindex e6e6becd6..e5b2ae23d 100644--- a/scrapy/utils/deprecate.py+++ b/scrapy/utils/deprecate.py@@ -203,7 +203,7 @@ def argument_is_required(func: Callable[..., Any], arg_name: str) -> bool:"""Check if a function argument is required (exists and doesn't have a default value).- .. versionadded:: VERSION+ .. versionadded:: 2.14>>> def func(a, b=1, c=None):... passdiff --git a/scrapy/utils/python.py b/scrapy/utils/python.pyindex ca4b4bf28..04da75a30 100644--- a/scrapy/utils/python.py+++ b/scrapy/utils/python.py@@ -174,7 +174,7 @@ def get_func_args_dict() -> Mapping[str, inspect.Parameter]:"""Return the argument dict of a callable object.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""if not callable(func):raise TypeError(f"func must be callable, got '{type(func).__name__}'")diff --git a/scrapy/utils/signal.py b/scrapy/utils/signal.pyindex 2c6aa9e67..a06936544 100644--- a/scrapy/utils/signal.py+++ b/scrapy/utils/signal.py@@ -142,7 +142,7 @@ async def send_catch_log_async(Returns a coroutine that completes once all signal handlers have finished.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""# note that this returns exceptions instead of Failures in the second tuple memberif is_asyncio_available():@@ -171,7 +171,7 @@ async def _send_catch_log_asyncio(:class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` to beinstalled.- .. versionadded:: VERSION+ .. versionadded:: 2.14"""dont_log = named.pop("dont_log", ())dont_log = tuple(dont_log) if isinstance(dont_log, Sequence) else (dont_log,)