Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Move old release notes into per-series pages (#8150)

Adrian03bdac819034
diff --git a/docs/conf.py b/docs/conf.py
index f2f0c2ce8..fa305025e 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -181,6 +181,7 @@ llms_txt_exclude = [
# Changelog, not useful for an LLM answering "how do I use Scrapy"
# questions, and the largest single contributor to llms-full.txt size.
"news.rst",
+ "news/*",
"contributing.rst",
]
diff --git a/docs/news.rst b/docs/news.rst
index 299358c8f..038915c36 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -2909,7620 +2909,31 @@ Quality assurance
:gh:`7008`,
:gh:`7177`)
-.. _release-2.13.4:
+Older releases
+--------------
-Scrapy 2.13.4 (2025-11-17)
---------------------------
-
-Security bug fixes
-~~~~~~~~~~~~~~~~~~
-
-- Improved protection against decompression bombs in
- :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
- for responses compressed using the ``br`` and ``deflate`` methods: if a
- single compressed chunk would be larger than the response size limit (see
- :setting:`DOWNLOAD_MAXSIZE`) when decompressed, decompression is no longer
- carried out. This is especially important for the ``br`` (Brotli) method
- that can provide a very high compression ratio. Please, see the
- `CVE-2025-6176`_ and `GHSA-2qfp-q593-8484`_ security advisories for more
- information.
- (:gh:`7134`)
-
- .. _CVE-2025-6176: https://nvd.nist.gov/vuln/detail/CVE-2025-6176
- .. _GHSA-2qfp-q593-8484: https://github.com/advisories/GHSA-2qfp-q593-8484
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- The minimum supported version of the optional ``brotli`` package is now
- ``1.2.0``.
- (:gh:`7134`)
-
-- The ``brotlicffi`` and ``brotlipy`` packages can no longer be used to
- decompress Brotli-compressed responses. Please install the ``brotli``
- package instead.
- (:gh:`7134`)
-
-Other changes
-~~~~~~~~~~~~~
-
-- Restricted the maximum supported Twisted version to ``25.5.0``, as Scrapy
- currently uses some private APIs changed in later Twisted versions.
- (:gh:`7142`)
-
-- Stopped setting the ``COVERAGE_CORE`` environment variable in tests, it
- didn't have an effect but caused the ``coverage`` module to produce a
- warning or an error.
- (:gh:`7137`)
-
-- Removed the documentation build dependency on the deprecated
- ``sphinx-hoverxref`` module.
- (:gh:`6786`, :gh:`6922`)
-
-.. _release-2.13.3:
-
-Scrapy 2.13.3 (2025-07-02)
---------------------------
-
-- Changed the values for :setting:`DOWNLOAD_DELAY` (from ``0`` to ``1``) and
- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (from ``8`` to ``1``) in the
- default project template.
- (:gh:`6597`, :gh:`6918`, :gh:`6923`)
-
-- Improved :class:`scrapy.core.engine.ExecutionEngine` logic related to
- initialization and exception handling, fixing several cases where the
- spider would crash, hang or log an unhandled exception.
- (:gh:`6783`, :gh:`6784`, :gh:`6900`, :gh:`6908`, :gh:`6910`,
- :gh:`6911`)
-
-- Fixed a Windows issue with :ref:`feed exports <topics-feed-exports>` using
- :class:`scrapy.extensions.feedexport.FileFeedStorage` that caused the file
- to be created on the wrong drive.
- (:gh:`6894`, :gh:`6897`)
-
-- Allowed running tests with Twisted 25.5.0+ again. Pytest 8.4.1+ is now
- required for running tests in non-pinned envs as support for the new
- Twisted version was added in that version.
- (:gh:`6893`)
-
-- Fixed running tests with lxml 6.0.0+.
- (:gh:`6919`)
-
-- Added a deprecation notice for
- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` to :ref:`the Scrapy
- 2.11.2 release notes <release-2.11.2>`.
- (:gh:`6926`)
-
-- Updated :ref:`contribution docs <topics-contributing>` to refer to ruff_
- instead of black_.
- (:gh:`6903`)
-
-- Added ``.venv/`` and ``.vscode/`` to ``.gitignore``.
- (:gh:`6901`, :gh:`6907`)
-
-
-.. _release-2.13.2:
-
-Scrapy 2.13.2 (2025-06-09)
---------------------------
-
-- Fixed a bug introduced in Scrapy 2.13.0 that caused results of request
- errbacks to be ignored when the errback was called because of a downloader
- error.
- (:gh:`6861`, :gh:`6863`)
-
-- Added a note about the behavior change of
- ``scrapy.utils.reactor.is_asyncio_reactor_installed`` to its docs and
- to the "Backward-incompatible changes" section of :ref:`the Scrapy 2.13.0
- release notes <release-2.13.0>`.
- (:gh:`6866`)
-
-- Improved the message in the exception raised by
- :func:`scrapy.utils.test.get_reactor_settings` when there is no reactor
- installed.
- (:gh:`6866`)
-
-- Updated the :class:`scrapy.crawler.CrawlerRunner` examples in
- :ref:`topics-practices` to install the reactor explicitly, to fix
- reactor-related errors with Scrapy 2.13.0 and later.
- (:gh:`6865`)
-
-- Fixed ``scrapy fetch`` not working with scrapy-poet_.
- (:gh:`6872`)
-
-- Fixed an exception produced by :class:`scrapy.core.engine.ExecutionEngine`
- when it's closed before being fully initialized.
- (:gh:`6857`, :gh:`6867`)
-
-- Improved the README, updated the Scrapy logo in it.
- (:gh:`6831`, :gh:`6833`, :gh:`6839`)
-
-- Restricted the Twisted version used in tests to below 25.5.0, as some tests
- fail with 25.5.0.
- (:gh:`6878`, :gh:`6882`)
-
-- Updated type hints for Twisted 25.5.0 changes.
- (:gh:`6882`)
-
-- Removed the old artwork.
- (:gh:`6874`)
-
-
-.. _release-2.13.1:
-
-Scrapy 2.13.1 (2025-05-28)
---------------------------
-
-- Give callback requests precedence over start requests when priority values
- are the same.
-
- This makes changes from 2.13.0 to start request handling more intuitive and
- backward compatible. For scenarios where all requests have the same
- priorities, in 2.13.0 all start requests were sent before the first
- callback request. In 2.13.1, same as in 2.12 and lower, start requests are
- only sent when there are not enough pending callback requests to reach
- concurrency limits.
-
- (:gh:`6828`)
-
-- Added a deepwiki_ badge to the README. (:gh:`6793`)
-
- .. _deepwiki: https://deepwiki.com/scrapy/scrapy
-
-- Fixed a typo in the code example of :ref:`start-requests-lazy`.
- (:gh:`6812`, :gh:`6815`)
-
-- Fixed a typo in the :ref:`coroutine-support` section of the documentation.
- (:gh:`6822`)
-
-- Made this page more prominently listed in PyPI project links.
- (:gh:`6826`)
-
-
-.. _release-2.13.0:
-
-Scrapy 2.13.0 (2025-05-08)
---------------------------
-
-Highlights:
-
-- The asyncio reactor is now enabled by default
-
-- Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`
- (async) and changed how it is iterated
-
-- Added the :reqmeta:`allow_offsite` request meta key
-
-- Spider middlewares that don't support asynchronous spider output are
- deprecated
-
-- Added a base class for :ref:`universal spider middlewares
- <universal-spider-middleware>`
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- Dropped support for PyPy 3.9.
- (:gh:`6613`)
-
-- Added support for PyPy 3.11.
- (:gh:`6697`)
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- The default value of the :setting:`TWISTED_REACTOR` setting was changed
- from ``None`` to
- ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
- was used in newly generated projects since Scrapy 2.7.0 but now existing
- projects that don't explicitly set this setting will also use the asyncio
- reactor. You can :ref:`change this setting in your project
- <disable-asyncio>` to use a different reactor.
- (:gh:`6659`, :gh:`6713`)
-
-- The iteration of start requests and items no longer stops once there are
- requests in the scheduler, and instead runs continuously until all start
- requests have been scheduled.
-
- To reproduce the previous behavior, see :ref:`start-requests-lazy`.
- (:gh:`6729`)
-
-- An unhandled exception from the
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.open_spider` method of a
- :ref:`spider middleware <topics-spider-middleware>` no longer stops the
- crawl.
- (:gh:`6729`)
-
-- In ``scrapy.core.engine.ExecutionEngine``:
-
- - The second parameter of ``open_spider()``, ``start_requests``, has been
- removed. The start requests are determined by the ``spider`` parameter
- instead (see :meth:`~scrapy.Spider.start`).
-
- - The ``slot`` attribute has been renamed to ``_slot`` and should not be
- used.
-
- (:gh:`6729`)
-
-- In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot``
- and should not be used.
- (:gh:`6729`)
-
-- The ``slot`` :ref:`telnet variable <telnet-vars>` has been removed.
- (:gh:`6729`)
-
-- In ``scrapy.core.spidermw.SpiderMiddlewareManager``,
- ``process_start_requests()`` has been replaced by ``process_start()``.
- (:gh:`6729`)
-
-- The ``scrape_func`` callable passed to
- ``scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response()`` is now
- called with 2 parameters, ``response`` and ``request``, instead of 3, and
- must return a :class:`~twisted.internet.defer.Deferred` instead of an
- iterable.
- (:gh:`6787`)
-
-- The now-deprecated ``start_requests()`` method, when it returns an iterable
- instead of being defined as a generator, is now executed *after* the
- :ref:`scheduler <topics-scheduler>` instance has been created.
- (:gh:`6729`)
-
-- When using :setting:`JOBDIR`, :ref:`start requests <start-requests>` are
- now serialized into their own, ``s``-suffixed priority folders. You can set
- :setting:`SCHEDULER_START_DISK_QUEUE` to ``None`` or ``""`` to change that,
- but the side effects may be undesirable. See
- :setting:`SCHEDULER_START_DISK_QUEUE` for details.
- (:gh:`6729`)
-
-- The URL length limit, set by the :setting:`URLLENGTH_LIMIT` setting, is now
- also enforced for start requests.
- (:gh:`6777`)
-
-- Calling ``scrapy.utils.reactor.is_asyncio_reactor_installed`` without
- an installed reactor now raises an exception instead of installing a
- reactor. This shouldn't affect normal Scrapy use cases, but it may affect
- 3rd-party test suites that use Scrapy internals such as
- :class:`~scrapy.crawler.Crawler` and don't install a reactor explicitly. If
- you are affected by this change, you most likely need to install the
- reactor before running Scrapy code that expects it to be installed.
- (:gh:`6732`, :gh:`6735`)
-
-- The ``from_settings()`` method of
- :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
- deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation
- period (this was needed because after the introduction of the
- :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
- switching built-in spider middlewares to it those middlewares need the
- :class:`~scrapy.crawler.Crawler` instance at run time). Please use
- ``from_crawler()`` instead.
- (:gh:`6693`)
-
-- ``scrapy.utils.url.escape_ajax()`` is no longer called when a
- :class:`~scrapy.Request` instance is created. It was only useful for
- websites supporting the ``_escaped_fragment_`` feature which most modern
- websites don't support. If you still need this you can modify the URLs
- before passing them to :class:`~scrapy.Request`.
- (:gh:`6523`, :gh:`6651`)
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- Removed old deprecated name aliases for some signals:
-
- - ``stats_spider_opened`` (use ``spider_opened`` instead)
-
- - ``stats_spider_closing`` and ``stats_spider_closed`` (use
- ``spider_closed`` instead)
-
- - ``item_passed`` (use ``item_scraped`` instead)
-
- - ``request_received`` (use ``request_scheduled`` instead)
-
- (:gh:`6654`, :gh:`6655`)
-
-Deprecations
-~~~~~~~~~~~~
-
-- The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated,
- use :meth:`~scrapy.Spider.start` instead, or both to maintain support for
- lower Scrapy versions.
- (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
-
-- The ``process_start_requests()`` method of :ref:`spider middlewares
- <topics-spider-middleware>` is deprecated, use
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
- or both to maintain support for lower Scrapy versions.
- (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
-
-- The ``__init__`` method of priority queue classes (see
- :setting:`SCHEDULER_PRIORITY_QUEUE`) should now support a keyword-only
- ``start_queue_cls`` parameter.
- (:gh:`6752`)
-
-- Spider middlewares that don't support asynchronous spider output are
- deprecated. The async iterable downgrading feature, needed for using such
- middlewares with asynchronous callbacks and with other spider middlewares
- that produce asynchronous iterables, is also deprecated. Please update all
- such middlewares to support asynchronous spider output. (:gh:`6664`)
-
-- Functions that were imported from :mod:`w3lib.url` and re-exported in
- :mod:`scrapy.utils.url` are now deprecated, you should import them from
- :mod:`w3lib.url` directly. They are:
-
- - ``scrapy.utils.url.add_or_replace_parameter()``
-
- - ``scrapy.utils.url.add_or_replace_parameters()``
-
- - ``scrapy.utils.url.any_to_uri()``
-
- - ``scrapy.utils.url.canonicalize_url()``
-
- - ``scrapy.utils.url.file_uri_to_path()``
-
- - ``scrapy.utils.url.is_url()``
-
- - ``scrapy.utils.url.parse_data_uri()``
-
- - ``scrapy.utils.url.parse_url()``
-
- - ``scrapy.utils.url.path_to_file_uri()``
-
- - ``scrapy.utils.url.safe_download_url()``
-
- - ``scrapy.utils.url.safe_url_string()``
-
- - ``scrapy.utils.url.url_query_cleaner()``
-
- - ``scrapy.utils.url.url_query_parameter()``
-
- (:gh:`4577`, :gh:`6583`, :gh:`6586`)
-
-- HTTP/1.0 support code is deprecated. It was disabled by default and
- couldn't be used together with HTTP/1.1. If you still need it, you should
- write your own download handler or copy the code from Scrapy. The
- deprecations include:
-
- - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
-
- - ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory``
-
- - ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter``
-
- - Overriding
- ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()``
-
- (:gh:`6634`)
-
-- The following modules and functions used only in tests are deprecated:
-
- - the ``scrapy.utils.testproc`` module
-
- - the ``scrapy.utils.testsite`` module
-
- - ``scrapy.utils.test.assert_gcs_environ()``
-
- - ``scrapy.utils.test.get_ftp_content_and_delete()``
-
- - ``scrapy.utils.test.get_gcs_content_and_delete()``
-
- - ``scrapy.utils.test.mock_google_cloud_storage()``
-
- - ``scrapy.utils.test.skip_if_no_boto()``
-
- If you need to use them in your tests or code, you can copy the code from Scrapy.
- (:gh:`6696`)
-
-- ``scrapy.utils.test.TestSpider`` is deprecated. If you need an empty spider
- class you can use :class:`scrapy.utils.spider.DefaultSpider` or create your
- own subclass of :class:`scrapy.Spider`.
- (:gh:`6678`)
-
-- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is
- deprecated. It was disabled by default and isn't useful for most of the
- existing websites.
- (:gh:`6523`, :gh:`6651`, :gh:`6656`)
-
-- ``scrapy.utils.url.escape_ajax()`` is deprecated.
- (:gh:`6523`, :gh:`6651`)
-
-- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,
- you can copy its code from Scrapy.
- (:gh:`6708`, :gh:`6714`)
-
-- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
- ``scrapy.utils.versions.get_versions()`` instead.
- (:gh:`6582`)
-
-- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be
- called.
- (:gh:`4151`)
-
-- Passing the ``spider`` argument to the following methods of
- :class:`~scrapy.core.scraper.Scraper` is deprecated:
-
- - ``close_spider()``
-
- - ``enqueue_scrape()``
-
- - ``handle_spider_error()``
-
- - ``handle_spider_output()``
-
- (:gh:`6764`)
-
-New features
-~~~~~~~~~~~~
-
-- You can now yield the start requests and items of a spider from the
- :meth:`~scrapy.Spider.start` spider method and from the
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` spider
- middleware method, both :term:`asynchronous generators <python:asynchronous
- generator>`.
-
- This makes it possible to use asynchronous code to generate those start
- requests and items, e.g. reading them from a queue service or database
- using an asynchronous client, without workarounds.
- (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
-
-- Start requests are now :ref:`scheduled <topics-scheduler>` as soon as
- possible.
-
- As a result, their :attr:`~scrapy.Request.priority` is now taken into
- account as soon as :setting:`CONCURRENT_REQUESTS` is reached.
- (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
-
-- :class:`Crawler.signals <scrapy.signalmanager.SignalManager>` has a new
- :meth:`~scrapy.signalmanager.SignalManager.wait_for` method.
- (:gh:`6729`)
-
-- Added a new :signal:`scheduler_empty` signal.
- (:gh:`6729`)
-
-- Added new settings: :setting:`SCHEDULER_START_DISK_QUEUE` and
- :setting:`SCHEDULER_START_MEMORY_QUEUE`.
- (:gh:`6729`)
-
-- Added :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware`, which
- sets :reqmeta:`is_start_request` to ``True`` on :ref:`start requests
- <start-requests>`.
- (:gh:`6729`)
-
-- Exposed a new method of :class:`Crawler.engine
- <scrapy.core.engine.ExecutionEngine>`:
- :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`.
- (:gh:`6729`)
-
-- Added the :reqmeta:`allow_offsite` request meta key that can be used
- instead of the more general :attr:`~scrapy.Request.dont_filter` request
- attribute to skip processing of the request by
- :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not
- by other code that checks :attr:`~scrapy.Request.dont_filter`).
- (:gh:`3690`, :gh:`6151`, :gh:`6366`)
-
-- Added an optional base class for spider middlewares,
- :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be
- helpful for writing :ref:`universal spider middlewares
- <universal-spider-middleware>` without boilerplate and code duplication.
- The built-in spider middlewares now inherit from this class.
- (:gh:`6693`, :gh:`6777`)
-
-- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called
- ``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
- <pre-crawler-settings>`.
- (:gh:`6544`, :gh:`6568`)
-
-- Added :ref:`helpers <priority-dict-helpers>` for modifying :ref:`component
- priority dictionary <component-priority-dictionaries>` settings.
- (:gh:`6614`)
-
-- Responses that use an unknown/unsupported encoding now produce a warning.
- If Scrapy knows that installing an additional package (such as brotli_)
- will allow decoding the response, that will be mentioned in the warning.
- (:gh:`4697`, :gh:`6618`)
-
-- Added the ``spider_exceptions/count`` stat which tracks the total count of
- exceptions (tracked also by per-type ``spider_exceptions/*`` stats).
- (:gh:`6739`, :gh:`6740`)
-
-- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
- :attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
- customizing the log level of the message that is logged when an item is
- dropped.
- (:gh:`6603`, :gh:`6608`)
-
-- Added support for the ``-b, --cookie`` curl argument to
- :meth:`scrapy.Request.from_curl`.
- (:gh:`6684`)
-
-- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
- list of software whose versions are logged when the spider starts.
- (:gh:`6582`)
-
-- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows
- disabling run time analysis of callback code used to warn about incorrect
- ``return`` statements in generator-based callbacks. You may need to disable
- this setting if this analysis breaks on your callback code.
- (:gh:`6731`, :gh:`6738`)
-
-Improvements
-~~~~~~~~~~~~
-
-- Removed or postponed some calls of :func:`itemadapter.is_item` to increase
- performance.
- (:gh:`6719`)
-
-- Improved the error message when running a ``scrapy`` command that requires
- a project (such as ``scrapy crawl``) outside of a project directory.
- (:gh:`2349`, :gh:`3426`)
-
-- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template
- for new projects.
- (:gh:`6587`)
-
-Bug fixes
-~~~~~~~~~
-
-- Yielding an item from :meth:`Spider.start <scrapy.Spider.start>` or from
- :meth:`SpiderMiddleware.process_start
- <scrapy.spidermiddlewares.SpiderMiddleware.process_start>` no longer delays
- the next iteration of starting requests and items by up to 5 seconds.
- (:gh:`6729`)
-
-- Fixed calculation of ``items_per_minute`` and ``responses_per_minute``
- stats.
- (:gh:`6599`)
-
-- Fixed an error initializing
- :class:`scrapy.extensions.feedexport.GCSFeedStorage`.
- (:gh:`6617`, :gh:`6628`)
-
-- Fixed an error running ``scrapy bench``.
- (:gh:`6632`, :gh:`6633`)
-
-- Fixed duplicated log messages about the reactor and the event loop.
- (:gh:`6636`, :gh:`6657`)
-
-- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at
- run time, required by tools such as scrapy-poet_.
- (:gh:`6665`, :gh:`6671`)
-
- .. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet
-
-- Calling ``scrapy.utils.reactor.is_asyncio_reactor_installed`` without
- an installed reactor now raises an exception instead of installing a
- reactor.
- (:gh:`6732`, :gh:`6735`)
-
-- Restored support for the ``x-gzip`` content encoding.
- (:gh:`6618`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Documented the setting values set in the default project template.
- (:gh:`6762`, :gh:`6775`)
-
-- Improved the docs about asynchronous iterable support in spider
- middlewares. (:gh:`6688`)
-
-- Improved the :ref:`docs <coroutine-deferred-apis>` about using
- :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
- code and included a list of such APIs.
- (:gh:`6677`, :gh:`6734`, :gh:`6776`)
-
-- Improved the :ref:`contribution docs <topics-contributing>`.
- (:gh:`6561`, :gh:`6575`)
-
-- Removed the ``Splash`` recommendation from the :ref:`headless browser
- <topics-headless-browsing>` suggestion. We no longer recommend using
- ``Splash`` and recommend using other headless browser solutions instead.
- (:gh:`6642`, :gh:`6701`)
-
-- Added the dark mode to the HTML documentation.
- (:gh:`6653`)
-
-- Other documentation improvements and fixes.
- (:gh:`4151`,
- :gh:`6526`,
- :gh:`6620`,
- :gh:`6621`,
- :gh:`6622`,
- :gh:`6623`,
- :gh:`6624`,
- :gh:`6721`,
- :gh:`6723`,
- :gh:`6780`)
-
-Packaging
-~~~~~~~~~
-
-- Switched from ``setup.py`` to ``pyproject.toml``.
- (:gh:`6514`, :gh:`6547`)
-
-- Switched the build backend from setuptools_ to hatchling_.
- (:gh:`6771`)
-
- .. _hatchling: https://pypi.org/project/hatchling/
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Replaced most linters with ruff_.
- (:gh:`6565`,
- :gh:`6576`,
- :gh:`6577`,
- :gh:`6581`,
- :gh:`6584`,
- :gh:`6595`,
- :gh:`6601`,
- :gh:`6631`)
-
- .. _ruff: https://docs.astral.sh/ruff/
-
-- Improved accuracy and performance of collecting test coverage.
- (:gh:`6255`, :gh:`6610`)
-
-- Fixed an error that prevented running tests from directories other than the
- top level source directory.
- (:gh:`6567`)
-
-- Reduced the amount of ``mockserver`` calls in tests to improve the overall
- test run time.
- (:gh:`6637`, :gh:`6648`)
-
-- Fixed tests that were running the same test code more than once.
- (:gh:`6646`, :gh:`6647`, :gh:`6650`)
-
-- Refactored tests to use more ``pytest`` features instead of ``unittest``
- ones where possible.
- (:gh:`6678`,
- :gh:`6680`,
- :gh:`6695`,
- :gh:`6699`,
- :gh:`6700`,
- :gh:`6702`,
- :gh:`6709`,
- :gh:`6710`,
- :gh:`6711`,
- :gh:`6712`,
- :gh:`6725`)
-
-- Type hints improvements and fixes.
- (:gh:`6578`,
- :gh:`6579`,
- :gh:`6593`,
- :gh:`6605`,
- :gh:`6694`)
-
-- CI and test improvements and fixes.
- (:gh:`5360`,
- :gh:`6271`,
- :gh:`6547`,
- :gh:`6560`,
- :gh:`6602`,
- :gh:`6607`,
- :gh:`6609`,
- :gh:`6613`,
- :gh:`6619`,
- :gh:`6626`,
- :gh:`6679`,
- :gh:`6703`,
- :gh:`6704`,
- :gh:`6716`,
- :gh:`6720`,
- :gh:`6722`,
- :gh:`6724`,
- :gh:`6741`,
- :gh:`6743`,
- :gh:`6766`,
- :gh:`6770`,
- :gh:`6772`,
- :gh:`6773`)
-
-- Code cleanups.
- (:gh:`6600`,
- :gh:`6606`,
- :gh:`6635`,
- :gh:`6764`)
-
-
-.. _release-2.12.0:
-
-Scrapy 2.12.0 (2024-11-18)
---------------------------
-
-Highlights:
-
-- Dropped support for Python 3.8, added support for Python 3.13
-
-- ``scrapy.Spider.start_requests()`` can now yield items
-
-- Added :class:`~scrapy.http.JsonResponse`
-
-- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- Dropped support for Python 3.8.
- (:gh:`6466`, :gh:`6472`)
-
-- Added support for Python 3.13.
- (:gh:`6166`)
-
-- Minimum versions increased for these dependencies:
-
- - Twisted_: 18.9.0 → 21.7.0
-
- - cryptography_: 36.0.0 → 37.0.0
-
- - pyOpenSSL_: 21.0.0 → 22.0.0
-
- - lxml_: 4.4.1 → 4.6.0
-
-- Removed ``setuptools`` from the dependency list.
- (:gh:`6487`)
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- User-defined cookies for HTTPS requests will have the ``secure`` flag set
- to ``True`` unless it's set to ``False`` explicitly. This is important when
- these cookies are reused in HTTP requests, e.g. after a redirect to an HTTP
- URL.
- (:gh:`6357`)
-
-- The Reppy-based ``robots.txt`` parser,
- ``scrapy.robotstxt.ReppyRobotParser``, was removed, as it doesn't support
- Python 3.9+.
- (:gh:`5230`, :gh:`6099`, :gh:`6499`)
-
-- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
- its subclasses was improved and it's possible that some previously working
- usage scenarios will no longer work. It can only affect you if you define
- custom subclasses of ``MediaPipeline`` or create instances of these
- pipelines via ``from_settings()`` or ``__init__()`` calls instead of
- ``from_crawler()`` calls.
-
- Previously, ``MediaPipeline.from_crawler()`` called the ``from_settings()``
- method if it existed or the ``__init__()`` method otherwise, and then did
- some additional initialization using the ``crawler`` instance. If the
- ``from_settings()`` method existed (like in ``FilesPipeline``) it called
- ``__init__()`` to create the instance. It wasn't possible to override
- ``from_crawler()`` without calling ``MediaPipeline.from_crawler()`` from it
- which, in turn, couldn't be called in some cases (including subclasses of
- ``FilesPipeline``).
-
- Now, in line with the general usage of ``from_crawler()`` and
- ``from_settings()`` and the deprecation of the latter the recommended
- initialization order is the following one:
-
- - All ``__init__()`` methods should take a ``crawler`` argument. If they
- also take a ``settings`` argument they should ignore it, using
- ``crawler.settings`` instead. When they call ``__init__()`` of the base
- class they should pass the ``crawler`` argument to it too.
- - A ``from_settings()`` method shouldn't be defined. Class-specific
- initialization code should go into either an overridden ``from_crawler()``
- method or into ``__init__()``.
- - It's now possible to override ``from_crawler()`` and it's not necessary
- to call ``MediaPipeline.from_crawler()`` in it if other recommendations
- were followed.
- - If pipeline instances were created with ``from_settings()`` or
- ``__init__()`` calls (which wasn't supported even before, as it missed
- important initialization code), they should now be created with
- ``from_crawler()`` calls.
-
- (:gh:`6540`)
-
-- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
- <scrapy.pipelines.images.ImagesPipeline.convert_image>` is now
- positional-only, as it was changed from optional to required.
- (:gh:`6500`)
-
-- The ``convert`` argument of :func:`scrapy.utils.conf.build_component_list`
- is now positional-only, as the preceding argument (``custom``) was removed.
- (:gh:`6500`)
-
-- The ``overwrite_output`` argument of
- :func:`scrapy.utils.conf.feed_process_params_from_cli` is now
- positional-only, as the preceding argument (``output_format``) was removed.
- (:gh:`6500`)
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- Removed the ``scrapy.utils.request.request_fingerprint()`` function,
- deprecated in Scrapy 2.7.0.
- (:gh:`6212`, :gh:`6213`)
-
-- Removed support for value ``"2.6"`` of setting
- ``REQUEST_FINGERPRINTER_IMPLEMENTATION``, deprecated in Scrapy 2.7.0.
- (:gh:`6212`, :gh:`6213`)
-
-- :class:`~scrapy.dupefilters.RFPDupeFilter` subclasses now require
- supporting the ``fingerprinter`` parameter in their ``__init__`` method,
- introduced in Scrapy 2.7.0.
- (:gh:`6102`, :gh:`6113`)
-
-- Removed the ``scrapy.downloadermiddlewares.decompression`` module,
- deprecated in Scrapy 2.7.0.
- (:gh:`6100`, :gh:`6113`)
-
-- Removed the ``scrapy.utils.response.response_httprepr()`` function,
- deprecated in Scrapy 2.6.0.
- (:gh:`6111`, :gh:`6116`)
-
-- Spiders with spider-level HTTP authentication, i.e. with the ``http_user``
- or ``http_pass`` attributes, must now define ``http_auth_domain`` as well,
- which was introduced in Scrapy 2.5.1.
- (:gh:`6103`, :gh:`6113`)
-
-- :ref:`Media pipelines <topics-media-pipeline>` methods ``file_path()``,
- ``file_downloaded()``, ``get_images()``, ``image_downloaded()``,
- ``media_downloaded()``, ``media_to_download()``, and ``thumb_path()`` must
- now support an ``item`` parameter, added in Scrapy 2.4.0.
- (:gh:`6107`, :gh:`6113`)
-
-- The ``__init__()`` and ``from_crawler()`` methods of :ref:`feed storage
- backend classes <topics-feed-storage>` must now support the keyword-only
- ``feed_options`` parameter, introduced in Scrapy 2.4.0.
- (:gh:`6105`, :gh:`6113`)
-
-- Removed the ``scrapy.loader.common`` and ``scrapy.loader.processors``
- modules, deprecated in Scrapy 2.3.0.
- (:gh:`6106`, :gh:`6113`)
-
-- Removed the ``scrapy.utils.misc.extract_regex()`` function, deprecated in
- Scrapy 2.3.0.
- (:gh:`6106`, :gh:`6113`)
-
-- Removed the ``scrapy.http.JSONRequest`` class, replaced with
- ``JsonRequest`` in Scrapy 1.8.0.
- (:gh:`6110`, :gh:`6113`)
-
-- ``scrapy.utils.log.logformatter_adapter`` no longer supports missing
- ``args``, ``level``, or ``msg`` parameters, and no longer supports a
- ``format`` parameter, all scenarios that were deprecated in Scrapy 1.0.0.
- (:gh:`6109`, :gh:`6116`)
-
-- A custom class assigned to the :setting:`SPIDER_LOADER_CLASS` setting that
- does not implement the ``ISpiderLoader`` interface
- will now raise a :exc:`zope.interface.verify.DoesNotImplement` exception at
- run time. Non-compliant classes have been triggering a deprecation warning
- since Scrapy 1.0.0.
- (:gh:`6101`, :gh:`6113`)
-
-- Removed the ``--output-format``/``-t`` command line option, deprecated in
- Scrapy 2.1.0. ``-O <URI>:<FORMAT>`` should be used instead.
- (:gh:`6500`)
-
-- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
- :class:`~scrapy.crawler.Crawler` instance, deprecated in Scrapy 2.11.0, now
- raises an exception.
- (:gh:`6500`)
-
-- Subclassing
- :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
- without support for the ``crawler`` argument in ``__init__()`` and without
- a custom ``from_crawler()`` method, deprecated in Scrapy 2.5.0, is no
- longer allowed.
- (:gh:`6500`)
-
-- Removed the ``EXCEPTIONS_TO_RETRY`` attribute of
- :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`, deprecated in
- Scrapy 2.10.0.
- (:gh:`6500`)
-
-- Removed support for :ref:`S3 feed exports <topics-feed-storage-s3>` without
- the boto3_ package installed, deprecated in Scrapy 2.10.0.
- (:gh:`6500`)
-
-- Removed the ``scrapy.extensions.feedexport._FeedSlot`` class, deprecated in
- Scrapy 2.10.0.
- (:gh:`6500`)
-
-- Removed the ``scrapy.pipelines.images.NoimagesDrop`` exception, deprecated
- in Scrapy 2.8.0.
- (:gh:`6500`)
-
-- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
- <scrapy.pipelines.images.ImagesPipeline.convert_image>` is now required,
- not passing it was deprecated in Scrapy 2.8.0.
- (:gh:`6500`)
-
-- Removed the ``custom`` argument of
- :func:`scrapy.utils.conf.build_component_list`, deprecated in Scrapy
- 2.10.0.
- (:gh:`6500`)
-
-- Removed the ``scrapy.utils.reactor.get_asyncio_event_loop_policy()``
- function, deprecated in Scrapy 2.9.0. Use :func:`asyncio.get_event_loop`
- and related standard library functions instead.
- (:gh:`6500`)
-
-Deprecations
-~~~~~~~~~~~~
-
-- The ``from_settings()`` methods of the :ref:`Scrapy components
- <topics-components>` that have them are now deprecated. ``from_crawler()``
- should now be used instead. Affected components:
-
- - :class:`scrapy.dupefilters.RFPDupeFilter`
- - :class:`scrapy.mail.MailSender`
- - :class:`scrapy.middleware.MiddlewareManager`
- - :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`
- - :class:`scrapy.pipelines.files.FilesPipeline`
- - :class:`scrapy.pipelines.images.ImagesPipeline`
- - :class:`scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`
-
- (:gh:`6540`)
-
-- It's now deprecated to have a ``from_settings()`` method but no
- ``from_crawler()`` method in 3rd-party :ref:`Scrapy components
- <topics-components>`. You can define a simple ``from_crawler()`` method
- that calls ``cls.from_settings(crawler.settings)`` to fix this if you don't
- want to refactor the code. Note that if you have a ``from_crawler()``
- method Scrapy will not call the ``from_settings()`` method so the latter
- can be removed.
- (:gh:`6540`)
-
-- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
- its subclasses was improved and some old usage scenarios are now deprecated
- (see also the "Backward-incompatible changes" section). Specifically:
-
- - It's deprecated to define an ``__init__()`` method that doesn't take a
- ``crawler`` argument.
- - It's deprecated to call an ``__init__()`` method without passing a
- ``crawler`` argument. If it's passed, it's also deprecated to pass a
- ``settings`` argument, which will be ignored anyway.
- - Calling ``from_settings()`` is deprecated, use ``from_crawler()``
- instead.
- - Overriding ``from_settings()`` is deprecated, override ``from_crawler()``
- instead.
-
- (:gh:`6540`)
-
-- The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting is now deprecated.
- (:gh:`6212`, :gh:`6213`)
-
-- The ``scrapy.utils.misc.create_instance()`` function is now deprecated, use
- :func:`scrapy.utils.misc.build_from_crawler` instead.
- (:gh:`5523`, :gh:`5884`, :gh:`6162`, :gh:`6169`, :gh:`6540`)
-
-- ``scrapy.core.downloader.Downloader._get_slot_key()`` is deprecated, use
- :meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
- (:gh:`6340`, :gh:`6352`)
-
-- ``scrapy.utils.defer.process_chain_both()`` is now deprecated.
- (:gh:`6397`)
-
-- ``scrapy.twisted_version`` is now deprecated, you should instead use
- :attr:`twisted.version` directly (but note that it's an
- ``incremental.Version`` object, not a tuple).
- (:gh:`6509`, :gh:`6512`)
-
-- ``scrapy.utils.python.flatten()`` and ``scrapy.utils.python.iflatten()``
- are now deprecated.
- (:gh:`6517`, :gh:`6519`)
-
-- ``scrapy.utils.python.equal_attributes()`` is now deprecated.
- (:gh:`6517`, :gh:`6519`)
-
-- ``scrapy.utils.request.request_authenticate()`` is now deprecated, you
- should instead just set the ``Authorization`` header directly.
- (:gh:`6517`, :gh:`6519`)
-
-- ``scrapy.utils.serialize.ScrapyJSONDecoder`` is now deprecated, it didn't
- contain any code since Scrapy 1.0.0.
- (:gh:`6517`, :gh:`6519`)
-
-- ``scrapy.utils.test.assert_samelines()`` is now deprecated.
- (:gh:`6517`, :gh:`6519`)
-
-- ``scrapy.extensions.feedexport.build_storage()`` is now deprecated. You can
- instead call the builder callable directly.
- (:gh:`6540`)
-
-- ``scrapy.utils.misc.md5sum()`` is now deprecated.
- (:gh:`6264`)
-
-New features
-~~~~~~~~~~~~
-
-- ``scrapy.Spider.start_requests()`` can now yield items.
- (:gh:`5289`, :gh:`6417`)
-
- .. note:: Some spider middlewares may need to be updated for Scrapy 2.12
- support before you can use them in combination with the ability to
- yield items from ``start_requests()``.
-
-- Added a new :class:`~scrapy.http.Response` subclass,
- :class:`~scrapy.http.JsonResponse`, for responses with a `JSON MIME type
- <https://mimesniff.spec.whatwg.org/#json-mime-type>`_.
- (:gh:`6069`, :gh:`6171`, :gh:`6174`)
-
-- The :class:`~scrapy.extensions.logstats.LogStats` extension now adds
- ``items_per_minute`` and ``responses_per_minute`` to the :ref:`stats
- <topics-stats>` when the spider closes.
- (:gh:`4110`, :gh:`4111`)
-
-- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM` which allows closing the
- spider if no items were scraped in a set amount of time.
- (:gh:`6434`)
-
-- User-defined cookies can now include the ``secure`` field.
- (:gh:`6357`)
-
-- Added component getters to :class:`~scrapy.crawler.Crawler`:
- :meth:`~scrapy.crawler.Crawler.get_addon`,
- :meth:`~scrapy.crawler.Crawler.get_downloader_middleware`,
- :meth:`~scrapy.crawler.Crawler.get_extension`,
- :meth:`~scrapy.crawler.Crawler.get_item_pipeline`,
- :meth:`~scrapy.crawler.Crawler.get_spider_middleware`.
- (:gh:`6181`)
-
-- Slot delay updates by the :ref:`AutoThrottle extension
- <topics-autothrottle>` based on response latencies can now be disabled for
- specific requests via the :reqmeta:`autothrottle_dont_adjust_delay` meta
- key.
- (:gh:`6246`, :gh:`6527`)
-
-- If :setting:`SPIDER_LOADER_WARN_ONLY` is set to ``True``,
- :class:`~scrapy.spiderloader.SpiderLoader` does not raise
- :exc:`SyntaxError` but emits a warning instead.
- (:gh:`6483`, :gh:`6484`)
-
-- Added support for multiple-compressed responses (ones with several
- encodings in the ``Content-Encoding`` header).
- (:gh:`5143`, :gh:`5964`, :gh:`6063`)
-
-- Added support for multiple standard values in :setting:`REFERRER_POLICY`.
- (:gh:`6381`)
-
-- Added support for brotlicffi_ (previously named brotlipy_). brotli_ is
- still recommended but only brotlicffi_ works on PyPy.
- (:gh:`6263`, :gh:`6269`)
-
- .. _brotlicffi: https://github.com/python-hyper/brotlicffi
-
-- Added :class:`~scrapy.contracts.default.MetadataContract` that sets the
- request meta.
- (:gh:`6468`, :gh:`6469`)
-
-Improvements
-~~~~~~~~~~~~
-
-- Extended the list of file extensions that
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- ignores by default.
- (:gh:`6074`, :gh:`6125`)
-
-- :func:`scrapy.utils.httpobj.urlparse_cached` is now used in more places
- instead of :func:`urllib.parse.urlparse`.
- (:gh:`6228`, :gh:`6229`)
-
-Bug fixes
-~~~~~~~~~
-
-- :class:`~scrapy.pipelines.media.MediaPipeline` is now an abstract class and
- its methods that were expected to be overridden in subclasses are now
- abstract methods.
- (:gh:`6365`, :gh:`6368`)
-
-- Fixed handling of invalid ``@``-prefixed lines in contract extraction.
- (:gh:`6383`, :gh:`6388`)
-
-- Importing ``scrapy.extensions.telnet`` no longer installs the default
- reactor.
- (:gh:`6432`)
-
-- Reduced log verbosity for dropped requests that was increased in 2.11.2.
- (:gh:`6433`, :gh:`6475`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Added ``SECURITY.md`` that documents the security policy.
- (:gh:`5364`, :gh:`6051`)
-
-- Example code for :ref:`running Scrapy from a script <run-from-script>` no
- longer imports ``twisted.internet.reactor`` at the top level, which caused
- problems with non-default reactors when this code was used unmodified.
- (:gh:`6361`, :gh:`6374`)
-
-- Documented the :class:`~scrapy.extensions.spiderstate.SpiderState`
- extension.
- (:gh:`6278`, :gh:`6522`)
-
-- Other documentation improvements and fixes.
- (:gh:`5920`,
- :gh:`6094`,
- :gh:`6177`,
- :gh:`6200`,
- :gh:`6207`,
- :gh:`6216`,
- :gh:`6223`,
- :gh:`6317`,
- :gh:`6328`,
- :gh:`6389`,
- :gh:`6394`,
- :gh:`6402`,
- :gh:`6411`,
- :gh:`6427`,
- :gh:`6429`,
- :gh:`6440`,
- :gh:`6448`,
- :gh:`6449`,
- :gh:`6462`,
- :gh:`6497`,
- :gh:`6506`,
- :gh:`6507`,
- :gh:`6524`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Added ``py.typed``, in line with `PEP 561
- <https://peps.python.org/pep-0561/>`_.
- (:gh:`6058`, :gh:`6059`)
-
-- Fully covered the code with type hints (except for the most complicated
- parts, mostly related to ``twisted.web.http`` and other Twisted parts
- without type hints).
- (:gh:`5989`,
- :gh:`6097`,
- :gh:`6127`,
- :gh:`6129`,
- :gh:`6130`,
- :gh:`6133`,
- :gh:`6143`,
- :gh:`6191`,
- :gh:`6268`,
- :gh:`6274`,
- :gh:`6275`,
- :gh:`6276`,
- :gh:`6279`,
- :gh:`6325`,
- :gh:`6326`,
- :gh:`6333`,
- :gh:`6335`,
- :gh:`6336`,
- :gh:`6337`,
- :gh:`6341`,
- :gh:`6353`,
- :gh:`6356`,
- :gh:`6370`,
- :gh:`6371`,
- :gh:`6384`,
- :gh:`6385`,
- :gh:`6387`,
- :gh:`6391`,
- :gh:`6395`,
- :gh:`6414`,
- :gh:`6422`,
- :gh:`6460`,
- :gh:`6466`,
- :gh:`6472`,
- :gh:`6494`,
- :gh:`6498`,
- :gh:`6516`)
-
-- Improved Bandit_ checks.
- (:gh:`6260`, :gh:`6264`, :gh:`6265`)
-
-- Added pyupgrade_ to the ``pre-commit`` configuration.
- (:gh:`6392`)
-
- .. _pyupgrade: https://github.com/asottile/pyupgrade
-
-- Added ``flake8-bugbear``, ``flake8-comprehensions``, ``flake8-debugger``,
- ``flake8-docstrings``, ``flake8-string-format`` and
- ``flake8-type-checking`` to the ``pre-commit`` configuration.
- (:gh:`6406`, :gh:`6413`)
-
-- CI and test improvements and fixes.
- (:gh:`5285`,
- :gh:`5454`,
- :gh:`5997`,
- :gh:`6078`,
- :gh:`6084`,
- :gh:`6087`,
- :gh:`6132`,
- :gh:`6153`,
- :gh:`6154`,
- :gh:`6201`,
- :gh:`6231`,
- :gh:`6232`,
- :gh:`6235`,
- :gh:`6236`,
- :gh:`6242`,
- :gh:`6245`,
- :gh:`6253`,
- :gh:`6258`,
- :gh:`6259`,
- :gh:`6270`,
- :gh:`6272`,
- :gh:`6286`,
- :gh:`6290`,
- :gh:`6296`
- :gh:`6367`,
- :gh:`6372`,
- :gh:`6403`,
- :gh:`6416`,
- :gh:`6435`,
- :gh:`6489`,
- :gh:`6501`,
- :gh:`6504`,
- :gh:`6511`,
- :gh:`6543`,
- :gh:`6545`)
-
-- Code cleanups.
- (:gh:`6196`,
- :gh:`6197`,
- :gh:`6198`,
- :gh:`6199`,
- :gh:`6254`,
- :gh:`6257`,
- :gh:`6285`,
- :gh:`6305`,
- :gh:`6343`,
- :gh:`6349`,
- :gh:`6386`,
- :gh:`6415`,
- :gh:`6463`,
- :gh:`6470`,
- :gh:`6499`,
- :gh:`6505`,
- :gh:`6510`,
- :gh:`6531`,
- :gh:`6542`)
-
-Other
-~~~~~
-
-- Issue tracker improvements. (:gh:`6066`)
-
-
-.. _release-2.11.2:
-
-Scrapy 2.11.2 (2024-05-14)
---------------------------
-
-Security bug fixes
-~~~~~~~~~~~~~~~~~~
-
-- Redirects to non-HTTP protocols are no longer followed. Please, see the
- `23j4-mw76-5v7h security advisory`_ for more information. (:gh:`457`)
-
- .. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h
-
-- The ``Authorization`` header is now dropped on redirects to a different
- scheme (``http://`` or ``https://``) or port, even if the domain is the
- same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more
- information.
-
- .. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f
-
-- When using system proxy settings that are different for ``http://`` and
- ``https://``, redirects to a different URL scheme will now also trigger the
- corresponding change in proxy settings for the redirected request. Please,
- see the `jm3v-qxmh-hxwv security advisory`_ for more information.
- (:gh:`767`)
-
- .. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv
-
-- :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>` is now
- enforced for all requests, and not only requests from spider callbacks.
- (:gh:`1042`, :gh:`2241`, :gh:`6358`)
-
-- :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML
- entities. (:gh:`6265`)
-
-- defusedxml_ is now used to make
- :class:`scrapy.http.request.rpc.XmlRpcRequest` more secure.
- (:gh:`6250`, :gh:`6251`)
-
- .. _defusedxml: https://github.com/tiran/defusedxml
-
-Deprecations
-~~~~~~~~~~~~
-
-- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` (a spider
- middleware) is now deprecated and not enabled by default. The new
- downloader middleware with the same functionality,
- :class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, is enabled
- instead.
- (:gh:`2241`, :gh:`6358`)
-
-
-Bug fixes
-~~~~~~~~~
-
-- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
- favor of brotli_. (:gh:`6261`)
-
- .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
- instead if you can.
-
-- Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This
- prevents
- :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from
- following redirects that would not be followed by web browsers with
- JavaScript enabled. (:gh:`6342`, :gh:`6347`)
-
-- During :ref:`feed export <topics-feed-exports>`, do not close the
- underlying file from :ref:`built-in post-processing plugins
- <builtin-plugins>`.
- (:gh:`5932`, :gh:`6178`, :gh:`6239`)
-
-- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- now properly applies the ``unique`` and ``canonicalize`` parameters.
- (:gh:`3273`, :gh:`6221`)
-
-- Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty
- string. (:gh:`6121`, :gh:`6124`)
-
-- Fix :attr:`Spider.logger <scrapy.Spider.logger>` not logging custom extra
- information. (:gh:`6323`, :gh:`6324`)
-
-- ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing
- the UTF-8-compatible (e.g. ASCII) parts of the document.
- (:gh:`6292`, :gh:`6298`)
-
-- :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an
- exception if ``default`` is ``None``.
- (:gh:`6308`, :gh:`6310`)
-
-- :class:`~scrapy.Selector` now uses
- :func:`scrapy.utils.response.get_base_url` to determine the base URL of a
- given :class:`~scrapy.http.Response`. (:gh:`6265`)
-
-- The :meth:`media_to_download` method of :ref:`media pipelines
- <topics-media-pipeline>` now logs exceptions before stripping them.
- (:gh:`5067`, :gh:`5068`)
-
-- When passing a callback to the :command:`parse` command, build the callback
- callable with the right signature.
- (:gh:`6182`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`.
- (:gh:`6203`, :gh:`6208`)
-
-- Document that :attr:`scrapy.Selector.type` can be ``"json"``.
- (:gh:`6328`, :gh:`6334`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Make builds reproducible. (:gh:`5019`, :gh:`6322`)
-
-- Packaging and test fixes.
- (:gh:`6286`, :gh:`6290`, :gh:`6312`, :gh:`6316`, :gh:`6344`)
-
-
-.. _release-2.11.1:
-
-Scrapy 2.11.1 (2024-02-14)
---------------------------
-
-Highlights:
-
-- Security bug fixes.
-
-- Support for Twisted >= 23.8.0.
-
-- Documentation improvements.
-
-Security bug fixes
-~~~~~~~~~~~~~~~~~~
-
-- Addressed `ReDoS vulnerabilities`_:
-
- - ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of
- :func:`~scrapy.utils.iterators.xmliter_lxml`, which
- :class:`~scrapy.spiders.XMLFeedSpider` now uses.
-
- To minimize the impact of this change on existing code,
- :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
- the node namespace with a prefix in the node name, and big files with
- highly nested trees when using libxml2 2.7+.
-
- - Fixed regular expressions in the implementation of the
- :func:`~scrapy.utils.response.open_in_browser` function.
-
- Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
-
- .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
- .. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9
-
-- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
- to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
- advisory`_ for more information.
-
- .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7
-
-- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
- deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
- removed.
-
-- The ``Authorization`` header is now dropped on redirects to a different
- domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
- information.
-
- .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- The Twisted dependency is no longer restricted to < 23.8.0. (:gh:`6024`,
- :gh:`6064`, :gh:`6142`)
-
-Bug fixes
-~~~~~~~~~
-
-- The OS signal handling code was refactored to no longer use private Twisted
- functions. (:gh:`6024`, :gh:`6064`, :gh:`6112`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
- changes made in the 2.11.0 release. (:gh:`6057`, :gh:`6147`)
-
-- Extended documentation for :attr:`.Request.meta`.
- (:gh:`5565`)
-
-- Fixed the :reqmeta:`dont_merge_cookies` documentation. (:gh:`5936`,
- :gh:`6077`)
-
-- Added a link to Zyte's export guides to the :ref:`feed exports
- <topics-feed-exports>` documentation. (:gh:`6183`)
-
-- Added a missing note about backward-incompatible changes in
- :class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes.
- (:gh:`6060`, :gh:`6081`)
-
-- Added a missing note about removing the deprecated
- ``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes.
- (:gh:`6056`, :gh:`6061`)
-
-- Other documentation improvements. (:gh:`6128`, :gh:`6144`,
- :gh:`6163`, :gh:`6190`, :gh:`6192`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Added Python 3.12 to the CI configuration, re-enabled tests that were
- disabled when the pre-release support was added. (:gh:`5985`,
- :gh:`6083`, :gh:`6098`)
-
-- Fixed a test issue on PyPy 7.3.14. (:gh:`6204`, :gh:`6205`)
-
-
-.. _release-2.11.0:
-
-Scrapy 2.11.0 (2023-09-18)
---------------------------
-
-Highlights:
-
-- Spiders can now modify :ref:`settings <topics-settings>` in their
- :meth:`~scrapy.Spider.from_crawler` methods, e.g. based on :ref:`spider
- arguments <spiderargs>`.
-
-- Periodic logging of stats.
-
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Most of the initialization of :class:`scrapy.crawler.Crawler` instances is
- now done in :meth:`~scrapy.crawler.Crawler.crawl`, so the state of
- instances before that method is called is now different compared to older
- Scrapy versions. We do not recommend using the
- :class:`~scrapy.crawler.Crawler` instances before
- :meth:`~scrapy.crawler.Crawler.crawl` is called. (:gh:`6038`)
-
-- :meth:`scrapy.Spider.from_crawler` is now called before the initialization
- of various components previously initialized in
- :meth:`scrapy.crawler.Crawler.__init__` and before the settings are
- finalized and frozen. This change was needed to allow changing the settings
- in :meth:`scrapy.Spider.from_crawler`. If you want to access the final
- setting values and the initialized :class:`~scrapy.crawler.Crawler`
- attributes in the spider code as early as possible you can do this in
- ``scrapy.Spider.start_requests()`` or in a handler of the
- :signal:`engine_started` signal. (:gh:`6038`)
-
-- The :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method now
- requires the response to be in a valid JSON encoding (UTF-8, UTF-16, or
- UTF-32). If you need to deal with JSON documents in an invalid encoding,
- use ``json.loads(response.text)`` instead. (:gh:`6016`)
-
-- :class:`~scrapy.exporters.PythonItemExporter` used the binary output by
- default but it no longer does. (:gh:`6006`, :gh:`6007`)
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- Removed the binary export mode of
- :class:`~scrapy.exporters.PythonItemExporter`, deprecated in Scrapy 1.1.0.
- (:gh:`6006`, :gh:`6007`)
-
- .. note:: If you are using this Scrapy version on Scrapy Cloud with a stack
- that includes an older Scrapy version and get a "TypeError:
- Unexpected options: binary" error, you may need to add
- ``scrapinghub-entrypoint-scrapy >= 0.14.1`` to your project
- requirements or switch to a stack that includes Scrapy 2.11.
-
-- Removed the ``CrawlerRunner.spiders`` attribute, deprecated in Scrapy
- 1.0.0, use :attr:`CrawlerRunner.spider_loader
- <scrapy.crawler.CrawlerRunner.spider_loader>` instead. (:gh:`6010`)
-
-- The :func:`scrapy.utils.response.response_httprepr` function, deprecated in
- Scrapy 2.6.0, has now been removed. (:gh:`6111`)
-
-Deprecations
-~~~~~~~~~~~~
-
-- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
- :class:`scrapy.crawler.Crawler` instance is now deprecated. (:gh:`1587`,
- :gh:`6040`)
-
-New features
-~~~~~~~~~~~~
-
-- Spiders can now modify settings in their
- :meth:`~scrapy.Spider.from_crawler` method, e.g. based on :ref:`spider
- arguments <spiderargs>`. (:gh:`1305`, :gh:`1580`, :gh:`2392`,
- :gh:`3663`, :gh:`6038`)
-
-- Added the :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension
- which can be enabled to log stats and/or their differences periodically.
- (:gh:`5926`)
-
-- Optimized the memory usage in :meth:`TextResponse.json
- <scrapy.http.TextResponse.json>` by removing unnecessary body decoding.
- (:gh:`5968`, :gh:`6016`)
-
-- Links to ``.webp`` files are now ignored by :ref:`link extractors
- <topics-link-extractors>`. (:gh:`6021`)
-
-Bug fixes
-~~~~~~~~~
-
-- Fixed logging enabled add-ons. (:gh:`6036`)
-
-- Fixed :class:`~scrapy.mail.MailSender` producing invalid message bodies
- when the ``charset`` argument is passed to
- :meth:`~scrapy.mail.MailSender.send`. (:gh:`5096`, :gh:`5118`)
-
-- Fixed an exception when accessing ``self.EXCEPTIONS_TO_RETRY`` from a
- subclass of :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
- (:gh:`6049`, :gh:`6050`)
-
-- :meth:`scrapy.settings.BaseSettings.getdictorlist`, used to parse
- :setting:`FEED_EXPORT_FIELDS`, now handles tuple values. (:gh:`6011`,
- :gh:`6013`)
-
-- Calls to ``datetime.utcnow()``, no longer recommended to be used, have been
- replaced with calls to ``datetime.now()`` with a timezone. (:gh:`6014`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Updated a deprecated function call in a pipeline example. (:gh:`6008`,
- :gh:`6009`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Extended typing hints. (:gh:`6003`, :gh:`6005`, :gh:`6031`,
- :gh:`6034`)
-
-- Pinned brotli_ to 1.0.9 for the PyPy tests as 1.1.0 breaks them.
- (:gh:`6044`, :gh:`6045`)
-
-- Other CI and pre-commit improvements. (:gh:`6002`, :gh:`6013`,
- :gh:`6046`)
-
-.. _release-2.10.1:
-
-Scrapy 2.10.1 (2023-08-30)
---------------------------
-
-Marked ``Twisted >= 23.8.0`` as unsupported. (:gh:`6024`, :gh:`6026`)
-
-.. _release-2.10.0:
-
-Scrapy 2.10.0 (2023-08-04)
---------------------------
-
-Highlights:
-
-- Added Python 3.12 support, dropped Python 3.7 support.
-
-- The new add-ons framework simplifies configuring 3rd-party components that
- support it.
-
-- Exceptions to retry can now be configured.
-
-- Many fixes and improvements for feed exports.
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- Dropped support for Python 3.7. (:gh:`5953`)
-
-- Added support for the upcoming Python 3.12. (:gh:`5984`)
-
-- Minimum versions increased for these dependencies:
-
- - lxml_: 4.3.0 → 4.4.1
-
- - cryptography_: 3.4.6 → 36.0.0
-
-- ``pkg_resources`` is no longer used. (:gh:`5956`, :gh:`5958`)
-
-- boto3_ is now recommended instead of botocore_ for exporting to S3.
- (:gh:`5833`).
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- The value of the :setting:`FEED_STORE_EMPTY` setting is now ``True``
- instead of ``False``. In earlier Scrapy versions empty files were created
- even when this setting was ``False`` (which was a bug that is now fixed),
- so the new default should keep the old behavior. (:gh:`872`,
- :gh:`5847`)
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
- returning ``None`` or modifying the ``params`` input parameter, deprecated
- in Scrapy 2.6, is no longer supported. (:gh:`5994`, :gh:`5996`)
-
-- The ``scrapy.utils.reqser`` module, deprecated in Scrapy 2.6, is removed.
- (:gh:`5994`, :gh:`5996`)
-
-- The ``scrapy.squeues`` classes ``PickleFifoDiskQueueNonRequest``,
- ``PickleLifoDiskQueueNonRequest``, ``MarshalFifoDiskQueueNonRequest``,
- and ``MarshalLifoDiskQueueNonRequest``, deprecated in
- Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5996`)
-
-- The property ``open_spiders`` and the methods ``has_capacity`` and
- ``schedule`` of :class:`scrapy.core.engine.ExecutionEngine`,
- deprecated in Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5998`)
-
-- Passing a ``spider`` argument to the
- :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`,
- :meth:`~scrapy.core.engine.ExecutionEngine.crawl` and
- :meth:`~scrapy.core.engine.ExecutionEngine.download` methods of
- :class:`scrapy.core.engine.ExecutionEngine`, deprecated in Scrapy 2.6, is
- no longer supported. (:gh:`5994`, :gh:`5998`)
-
-Deprecations
-~~~~~~~~~~~~
-
-- :class:`scrapy.utils.datatypes.CaselessDict` is deprecated, use
- :class:`scrapy.utils.datatypes.CaseInsensitiveDict` instead.
- (:gh:`5146`)
-
-- Passing the ``custom`` argument to
- :func:`scrapy.utils.conf.build_component_list` is deprecated, it was used
- in the past to merge ``FOO`` and ``FOO_BASE`` setting values but now Scrapy
- uses :func:`scrapy.settings.BaseSettings.getwithbase` to do the same.
- Code that uses this argument and cannot be switched to ``getwithbase()``
- can be switched to merging the values explicitly. (:gh:`5726`,
- :gh:`5923`)
-
-New features
-~~~~~~~~~~~~
-
-- Added support for :ref:`Scrapy add-ons <topics-addons>`. (:gh:`5950`)
-
-- Added the :setting:`RETRY_EXCEPTIONS` setting that configures which
- exceptions will be retried by
- :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
- (:gh:`2701`, :gh:`5929`)
-
-- Added the possiiblity to close the spider if no items were produced in the
- specified time, configured by :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`.
- (:gh:`5979`)
-
-- Added support for the :setting:`AWS_REGION_NAME` setting to feed exports.
- (:gh:`5980`)
-
-- Added support for using :class:`pathlib.Path` objects that refer to
- absolute Windows paths in the :setting:`FEEDS` setting. (:gh:`5939`)
-
-Bug fixes
-~~~~~~~~~
-
-- Fixed creating empty feeds even with ``FEED_STORE_EMPTY=False``.
- (:gh:`872`, :gh:`5847`)
-
-- Fixed using absolute Windows paths when specifying output files.
- (:gh:`5969`, :gh:`5971`)
-
-- Fixed problems with uploading large files to S3 by switching to multipart
- uploads (requires boto3_). (:gh:`960`, :gh:`5735`, :gh:`5833`)
-
-- Fixed the JSON exporter writing extra commas when some exceptions occur.
- (:gh:`3090`, :gh:`5952`)
-
-- Fixed the "read of closed file" error in the CSV exporter. (:gh:`5043`,
- :gh:`5705`)
-
-- Fixed an error when a component added by the class object throws
- :exc:`~scrapy.exceptions.NotConfigured` with a message. (:gh:`5950`,
- :gh:`5992`)
-
-- Added the missing :meth:`scrapy.settings.BaseSettings.pop` method.
- (:gh:`5959`, :gh:`5960`, :gh:`5963`)
-
-- Added :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` as a replacement
- for :class:`~scrapy.utils.datatypes.CaselessDict` that fixes some API
- inconsistencies. (:gh:`5146`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Documented :meth:`scrapy.Spider.update_settings`. (:gh:`5745`,
- :gh:`5846`)
-
-- Documented possible problems with early Twisted reactor installation and
- their solutions. (:gh:`5981`, :gh:`6000`)
-
-- Added examples of making additional requests in callbacks. (:gh:`5927`)
-
-- Improved the feed export docs. (:gh:`5579`, :gh:`5931`)
-
-- Clarified the docs about request objects on redirection. (:gh:`5707`,
- :gh:`5937`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Added support for running tests against the installed Scrapy version.
- (:gh:`4914`, :gh:`5949`)
-
-- Extended typing hints. (:gh:`5925`, :gh:`5977`)
-
-- Fixed the ``test_utils_asyncio.AsyncioTest.test_set_asyncio_event_loop``
- test. (:gh:`5951`)
-
-- Fixed the ``test_feedexport.BatchDeliveriesTest.test_batch_path_differ``
- test on Windows. (:gh:`5847`)
-
-- Enabled CI runs for Python 3.11 on Windows. (:gh:`5999`)
-
-- Simplified skipping tests that depend on ``uvloop``. (:gh:`5984`)
-
-- Fixed the ``extra-deps-pinned`` tox env. (:gh:`5948`)
-
-- Implemented cleanups. (:gh:`5965`, :gh:`5986`)
-
-.. _release-2.9.0:
-
-Scrapy 2.9.0 (2023-05-08)
--------------------------
-
-Highlights:
-
-- Per-domain download settings.
-- Compatibility with new cryptography_ and new parsel_.
-- JMESPath selectors from the new parsel_.
-- Bug fixes.
-
-Deprecations
-~~~~~~~~~~~~
-
-- :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to
- :class:`scrapy.extensions.feedexport.FeedSlot` and the old name is
- deprecated. (:gh:`5876`)
-
-New features
-~~~~~~~~~~~~
-
-- Settings corresponding to :setting:`DOWNLOAD_DELAY`,
- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
- ``RANDOMIZE_DOWNLOAD_DELAY`` can now be set on a per-domain basis
- via the new :setting:`DOWNLOAD_SLOTS` setting. (:gh:`5328`)
-
-- Added :meth:`.TextResponse.jmespath`, a shortcut for JMESPath selectors
- available since parsel_ 1.8.1. (:gh:`5894`, :gh:`5915`)
-
-- Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
- signals. (:gh:`5876`)
-
-- Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a
- curl command from a :class:`~scrapy.Request` object. (:gh:`5892`)
-
-- Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be
- :class:`pathlib.Path` instances. (:gh:`5801`)
-
-Bug fixes
-~~~~~~~~~
-
-- Fixed a warning with Parsel 1.8.1+. (:gh:`5903`, :gh:`5918`)
-
-- Fixed an error when using feed postprocessing with S3 storage.
- (:gh:`5500`, :gh:`5581`)
-
-- Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method.
- (:gh:`5811`, :gh:`5821`)
-
-- Fixed an error when using cryptography_ 40.0.0+ and
- :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled.
- (:gh:`5857`, :gh:`5858`)
-
-- The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline`
- for files on Google Cloud Storage are no longer Base64-encoded.
- (:gh:`5874`, :gh:`5891`)
-
-- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed
- string values for the curl ``--data-raw`` argument, which are produced by
- browsers for data that includes certain symbols. (:gh:`5899`,
- :gh:`5901`)
-
-- The :command:`parse` command now also works with async generator callbacks.
- (:gh:`5819`, :gh:`5824`)
-
-- The :command:`genspider` command now properly works with HTTPS URLs.
- (:gh:`3553`, :gh:`5808`)
-
-- Improved handling of asyncio loops. (:gh:`5831`, :gh:`5832`)
-
-- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- now skips certain malformed URLs instead of raising an exception.
- (:gh:`5881`)
-
-- :func:`scrapy.utils.python.get_func_args` now supports more types of
- callables. (:gh:`5872`, :gh:`5885`)
-
-- Fixed an error when processing non-UTF8 values of ``Content-Type`` headers.
- (:gh:`5914`, :gh:`5917`)
-
-- Fixed an error breaking user handling of send failures in
- :meth:`scrapy.mail.MailSender.send`. (:gh:`1611`, :gh:`5880`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Expanded contributing docs. (:gh:`5109`, :gh:`5851`)
-
-- Added blacken-docs_ to pre-commit and reformatted the docs with it.
- (:gh:`5813`, :gh:`5816`)
-
-- Fixed a JS issue. (:gh:`5875`, :gh:`5877`)
-
-- Fixed ``make htmlview``. (:gh:`5878`, :gh:`5879`)
-
-- Fixed typos and other small errors. (:gh:`5827`, :gh:`5839`,
- :gh:`5883`, :gh:`5890`, :gh:`5895`, :gh:`5904`)
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Extended typing hints. (:gh:`5805`, :gh:`5889`, :gh:`5896`)
-
-- Tests for most of the examples in the docs are now run as a part of CI,
- found problems were fixed. (:gh:`5816`, :gh:`5826`, :gh:`5919`)
-
-- Removed usage of deprecated Python classes. (:gh:`5849`)
-
-- Silenced ``include-ignored`` warnings from coverage. (:gh:`5820`)
-
-- Fixed a random failure of the ``test_feedexport.test_batch_path_differ``
- test. (:gh:`5855`, :gh:`5898`)
-
-- Updated docstrings to match output produced by parsel_ 1.8.1 so that they
- don't cause test failures. (:gh:`5902`, :gh:`5919`)
-
-- Other CI and pre-commit improvements. (:gh:`5802`, :gh:`5823`,
- :gh:`5908`)
-
-.. _blacken-docs: https://github.com/adamchainz/blacken-docs
-
-.. _release-2.8.0:
-
-Scrapy 2.8.0 (2023-02-02)
--------------------------
-
-This is a maintenance release, with minor features, bug fixes, and cleanups.
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
- been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
- :class:`~gzip.GzipFile` instead.
- (:gh:`5719`)
-
-- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
- 2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
- instead.
- (:gh:`5719`)
-
-- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
- 2.0, has now been removed. Use
- :meth:`~scrapy.utils.python.MutableChain.__next__` instead.
- (:gh:`5719`)
-
-- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
- in Scrapy 2.0, has now been removed. Use
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- instead.
- (:gh:`5720`)
-
-- Support for using environment variables prefixed with ``SCRAPY_`` to
- override settings, deprecated in Scrapy 2.0, has now been removed.
- (:gh:`5724`)
-
-- Support for the ``noconnect`` query string argument in proxy URLs,
- deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
- to need it to work fine without it.
- (:gh:`5731`)
-
-- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
- 2.3, has now been removed.
- (:gh:`5719`)
-
-- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
- has now been removed.
- (:gh:`5719`)
-
-- The ``scrapy.utils.boto.is_botocore()`` function, deprecated in Scrapy 2.4,
- has now been removed.
- (:gh:`5719`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
- (:gh:`5368`, :gh:`5489`)
-
-- :meth:`ImagesPipeline.convert_image
- <scrapy.pipelines.images.ImagesPipeline.convert_image>` must now accept a
- ``response_body`` parameter.
- (:gh:`3055`, :gh:`3689`, :gh:`4753`)
-
-
-New features
-~~~~~~~~~~~~
-
-- Applied black_ coding style to files generated with the
- :command:`genspider` and :command:`startproject` commands.
- (:gh:`5809`, :gh:`5814`)
-
- .. _black: https://black.readthedocs.io/en/stable/
-
-- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
- ``settings.py`` file that the :command:`startproject` command generates.
- With this value, JSON exports won’t force the use of escape sequences for
- non-ASCII characters.
- (:gh:`5797`, :gh:`5800`)
-
-- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
- peak memory usage during checks, and the binary unit MiB is now used to
- avoid confusion.
- (:gh:`5717`, :gh:`5722`, :gh:`5727`)
-
-- The ``callback`` parameter of :class:`~scrapy.Request` can now be set
- to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
- ``None``, as the latter indicates that the default spider callback
- (:meth:`~scrapy.Spider.parse`) is to be used.
- (:gh:`5798`)
-
-
-Bug fixes
-~~~~~~~~~
-
-- Enabled unsafe legacy SSL renegotiation to fix access to some outdated
- websites.
- (:gh:`5491`, :gh:`5790`)
-
-- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
- better.
- (:gh:`5386`, :gh:`5406`)
-
-- Fixed the :meth:`finish_exporting` method of :ref:`item exporters
- <topics-exporters>` not being called for empty files.
- (:gh:`5537`, :gh:`5758`)
-
-- Fixed HTTP/2 responses getting only the last value for a header when
- multiple headers with the same name are received.
- (:gh:`5777`)
-
-- Fixed an exception raised by the :command:`shell` command on some cases
- when :ref:`using asyncio <using-asyncio>`.
- (:gh:`5740`, :gh:`5742`, :gh:`5748`, :gh:`5759`, :gh:`5760`,
- :gh:`5771`)
-
-- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
- (``cb_kwargs``) added to a request in the ``process_request`` callback of a
- :class:`~scrapy.spiders.Rule` will no longer be ignored.
- (:gh:`5699`)
-
-- The :ref:`images pipeline <images-pipeline>` no longer re-encodes JPEG
- files.
- (:gh:`3055`, :gh:`3689`, :gh:`4753`)
-
-- Fixed the handling of transparent WebP images by the :ref:`images pipeline
- <images-pipeline>`.
- (:gh:`3072`, :gh:`5766`, :gh:`5767`)
-
-- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
- (Ctrl+C).
- (:gh:`2918`)
-
-- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- with ``unique=False`` no longer filters out links that have identical URL
- *and* text.
- (:gh:`3798`, :gh:`3799`, :gh:`4695`, :gh:`5458`)
-
-- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
- ignores URL protocols that do not support ``robots.txt`` (``data://``,
- ``file://``).
- (:gh:`5807`)
-
-- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
- (:gh:`5753`, :gh:`5754`)
-
-- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
- line.
- (:gh:`5709`, :gh:`5711`, :gh:`5712`)
-
-- Made the active project indication in the output of :ref:`commands
- <topics-commands>` more clear.
- (:gh:`5715`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-- Documented how to :ref:`debug spiders from Visual Studio Code
- <debug-vscode>`.
- (:gh:`5721`)
-
-- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
- (:gh:`5083`, :gh:`5540`)
-
-- Improved consistency.
- (:gh:`5761`)
-
-- Fixed typos.
- (:gh:`5714`, :gh:`5744`, :gh:`5764`)
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Applied :ref:`black coding style <coding-style>`, sorted import statements,
- and introduced :ref:`pre-commit <scrapy-pre-commit>`.
- (:gh:`4654`, :gh:`4658`, :gh:`5734`, :gh:`5737`, :gh:`5806`,
- :gh:`5810`)
-
-- Switched from :mod:`os.path` to :mod:`pathlib`.
- (:gh:`4916`, :gh:`4497`, :gh:`5682`)
-
-- Addressed many issues reported by Pylint.
- (:gh:`5677`)
-
-- Improved code readability.
- (:gh:`5736`)
-
-- Improved package metadata.
- (:gh:`5768`)
-
-- Removed direct invocations of ``setup.py``.
- (:gh:`5774`, :gh:`5776`)
-
-- Removed unnecessary :class:`~collections.OrderedDict` usages.
- (:gh:`5795`)
-
-- Removed unnecessary ``__str__`` definitions.
- (:gh:`5150`)
-
-- Removed obsolete code and comments.
- (:gh:`5725`, :gh:`5729`, :gh:`5730`, :gh:`5732`)
-
-- Fixed test and CI issues.
- (:gh:`5749`, :gh:`5750`, :gh:`5756`, :gh:`5762`, :gh:`5765`,
- :gh:`5780`, :gh:`5781`, :gh:`5782`, :gh:`5783`, :gh:`5785`,
- :gh:`5786`)
-
-
-.. _release-2.7.1:
-
-Scrapy 2.7.1 (2022-11-02)
--------------------------
-
-New features
-~~~~~~~~~~~~
-
-- Relaxed the restriction introduced in 2.6.2 so that the
- ``Proxy-Authorization`` header can again be set explicitly, as long as the
- proxy URL in the :reqmeta:`proxy` metadata has no other credentials, and
- for as long as that proxy URL remains the same; this restores compatibility
- with scrapy-zyte-smartproxy 2.1.0 and older (:gh:`5626`).
-
-Bug fixes
-~~~~~~~~~
-
-- Using ``-O``/``--overwrite-output`` and ``-t``/``--output-format`` options
- together now produces an error instead of ignoring the former option
- (:gh:`5516`, :gh:`5605`).
-
-- Replaced deprecated :mod:`asyncio` APIs that implicitly use the current
- event loop with code that explicitly requests a loop from the event loop
- policy (:gh:`5685`, :gh:`5689`).
-
-- Fixed uses of deprecated Scrapy APIs in Scrapy itself (:gh:`5588`,
- :gh:`5589`).
-
-- Fixed uses of a deprecated Pillow API (:gh:`5684`, :gh:`5692`).
-
-- Improved code that checks if generators return values, so that it no longer
- fails on decorated methods and partial methods (:gh:`5323`,
- :gh:`5592`, :gh:`5599`, :gh:`5691`).
-
-Documentation
-~~~~~~~~~~~~~
-
-- Upgraded the Code of Conduct to Contributor Covenant v2.1 (:gh:`5698`).
-
-- Fixed typos (:gh:`5681`, :gh:`5694`).
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Re-enabled some erroneously disabled flake8 checks (:gh:`5688`).
-
-- Ignored harmless deprecation warnings from :mod:`typing` in tests
- (:gh:`5686`, :gh:`5697`).
-
-- Modernized our CI configuration (:gh:`5695`, :gh:`5696`).
-
-
-.. _release-2.7.0:
-
-Scrapy 2.7.0 (2022-10-17)
------------------------------
-
-Highlights:
-
-- Added Python 3.11 support, dropped Python 3.6 support
-- Improved support for :ref:`asynchronous callbacks <topics-coroutines>`
-- :ref:`Asyncio support <using-asyncio>` is enabled by default on new
- projects
-- Output names of item fields can now be arbitrary strings
-- Centralized :ref:`request fingerprinting <request-fingerprints>`
- configuration is now possible
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-Python 3.7 or greater is now required; support for Python 3.6 has been dropped.
-Support for the upcoming Python 3.11 has been added.
-
-The minimum required version of some dependencies has changed as well:
-
-- lxml_: 3.5.0 → 4.3.0
-
-- Pillow_ (:ref:`images pipeline <images-pipeline>`): 4.0.0 → 7.1.0
-
-- zope.interface_: 5.0.0 → 5.1.0
-
-(:gh:`5512`, :gh:`5514`, :gh:`5524`, :gh:`5563`, :gh:`5664`,
-:gh:`5670`, :gh:`5678`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-- :meth:`ImagesPipeline.thumb_path
- <scrapy.pipelines.images.ImagesPipeline.thumb_path>` must now accept an
- ``item`` parameter (:gh:`5504`, :gh:`5508`).
-
-- The ``scrapy.downloadermiddlewares.decompression`` module is now
- deprecated (:gh:`5546`, :gh:`5547`).
-
-
-New features
-~~~~~~~~~~~~
-
-- The
- :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
- method of :ref:`spider middlewares <topics-spider-middleware>` can now be
- defined as an :term:`asynchronous generator` (:gh:`4978`).
-
-- The output of :class:`~scrapy.Request` callbacks defined as
- :ref:`coroutines <topics-coroutines>` is now processed asynchronously
- (:gh:`4978`).
-
-- :class:`~scrapy.spiders.crawl.CrawlSpider` now supports :ref:`asynchronous
- callbacks <topics-coroutines>` (:gh:`5657`).
-
-- New projects created with the :command:`startproject` command have
- :ref:`asyncio support <using-asyncio>` enabled by default (:gh:`5590`,
- :gh:`5679`).
-
-- The :setting:`FEED_EXPORT_FIELDS` setting can now be defined as a
- dictionary to customize the output name of item fields, lifting the
- restriction that required output names to be valid Python identifiers, e.g.
- preventing them to have whitespace (:gh:`1008`, :gh:`3266`,
- :gh:`3696`).
-
-- You can now customize :ref:`request fingerprinting <request-fingerprints>`
- through the new :setting:`REQUEST_FINGERPRINTER_CLASS` setting, instead of
- having to change it on every Scrapy component that relies on request
- fingerprinting (:gh:`900`, :gh:`3420`, :gh:`4113`, :gh:`4762`,
- :gh:`4524`).
-
-- ``jsonl`` is now supported and encouraged as a file extension for `JSON
- Lines`_ files (:gh:`4848`).
-
- .. _JSON Lines: https://jsonlines.org/
-
-- :meth:`ImagesPipeline.thumb_path
- <scrapy.pipelines.images.ImagesPipeline.thumb_path>` now receives the
- source :ref:`item <topics-items>` (:gh:`5504`, :gh:`5508`).
-
-
-Bug fixes
-~~~~~~~~~
-
-- When using Google Cloud Storage with a :ref:`media pipeline
- <topics-media-pipeline>`, :setting:`FILES_EXPIRES` now also works when
- :setting:`FILES_STORE` does not point at the root of your Google Cloud
- Storage bucket (:gh:`5317`, :gh:`5318`).
-
-- The :command:`parse` command now supports :ref:`asynchronous callbacks
- <topics-coroutines>` (:gh:`5424`, :gh:`5577`).
-
-- When using the :command:`parse` command with a URL for which there is no
- available spider, an exception is no longer raised (:gh:`3264`,
- :gh:`3265`, :gh:`5375`, :gh:`5376`, :gh:`5497`).
-
-- :class:`~scrapy.http.TextResponse` now gives higher priority to the `byte
- order mark`_ when determining the text encoding of the response body,
- following the `HTML living standard`_ (:gh:`5601`, :gh:`5611`).
-
- .. _byte order mark: https://en.wikipedia.org/wiki/Byte_order_mark
- .. _HTML living standard: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
-
-- MIME sniffing takes the response body into account in FTP and HTTP/1.0
- requests, as well as in cached requests (:gh:`4873`).
-
-- MIME sniffing now detects valid HTML 5 documents even if the ``html`` tag
- is missing (:gh:`4873`).
-
-- An exception is now raised if :setting:`ASYNCIO_EVENT_LOOP` has a value
- that does not match the asyncio event loop actually installed
- (:gh:`5529`).
-
-- Fixed :meth:`Headers.getlist() <scrapy.http.headers.Headers.getlist>`
- returning only the last header (:gh:`5515`, :gh:`5526`).
-
-- Fixed :class:`LinkExtractor
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` not ignoring the
- ``tar.gz`` file extension by default (:gh:`1837`, :gh:`2067`,
- :gh:`4066`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-- Clarified the return type of :meth:`Spider.parse <scrapy.Spider.parse>`
- (:gh:`5602`, :gh:`5608`).
-
-- To enable
- :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
- to do `brotli compression`_, installing brotli_ is now recommended instead
- of installing brotlipy_, as the former provides a more recent version of
- brotli.
-
- .. _brotli: https://github.com/google/brotli
- .. _brotli compression: https://www.ietf.org/rfc/rfc7932.txt
-
-- :ref:`Signal documentation <topics-signals>` now mentions :ref:`coroutine
- support <topics-coroutines>` and uses it in code examples (:gh:`4852`,
- :gh:`5358`).
-
-- :ref:`bans` now recommends `Common Crawl`_ instead of `Google cache`_
- (:gh:`3582`, :gh:`5432`).
-
- .. _Common Crawl: https://commoncrawl.org/
- .. _Google cache: https://www.googleguide.com/cached_pages.html
-
-- The new :ref:`topics-components` topic covers enforcing requirements on
- Scrapy components, like :ref:`downloader middlewares
- <topics-downloader-middleware>`, :ref:`extensions <topics-extensions>`,
- :ref:`item pipelines <topics-item-pipeline>`, :ref:`spider middlewares
- <topics-spider-middleware>`, and more; :ref:`enforce-asyncio-requirement`
- has also been added (:gh:`4978`).
-
-- :ref:`topics-settings` now indicates that setting values must be
- :ref:`picklable <pickle-picklable>` (:gh:`5607`, :gh:`5629`).
-
-- Removed outdated documentation (:gh:`5446`, :gh:`5373`,
- :gh:`5369`, :gh:`5370`, :gh:`5554`).
-
-- Fixed typos (:gh:`5442`, :gh:`5455`, :gh:`5457`, :gh:`5461`,
- :gh:`5538`, :gh:`5553`, :gh:`5558`, :gh:`5624`, :gh:`5631`).
-
-- Fixed other issues (:gh:`5283`, :gh:`5284`, :gh:`5559`,
- :gh:`5567`, :gh:`5648`, :gh:`5659`, :gh:`5665`).
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-- Added a continuous integration job to run `twine check`_ (:gh:`5655`,
- :gh:`5656`).
-
- .. _twine check: https://twine.readthedocs.io/en/stable/#twine-check
-
-- Addressed test issues and warnings (:gh:`5560`, :gh:`5561`,
- :gh:`5612`, :gh:`5617`, :gh:`5639`, :gh:`5645`, :gh:`5662`,
- :gh:`5671`, :gh:`5675`).
-
-- Cleaned up code (:gh:`4991`, :gh:`4995`, :gh:`5451`,
- :gh:`5487`, :gh:`5542`, :gh:`5667`, :gh:`5668`, :gh:`5672`).
-
-- Applied minor code improvements (:gh:`5661`).
-
-
-.. _release-2.6.3:
-
-Scrapy 2.6.3 (2022-09-27)
--------------------------
-
-- Added support for pyOpenSSL_ 22.1.0, removing support for SSLv3
- (:gh:`5634`, :gh:`5635`, :gh:`5636`).
-
-- Upgraded the minimum versions of the following dependencies:
-
- - cryptography_: 2.0 → 3.3
-
- - pyOpenSSL_: 16.2.0 → 21.0.0
-
- - service_identity_: 16.0.0 → 18.1.0
-
- - Twisted_: 17.9.0 → 18.9.0
-
- - zope.interface_: 4.1.3 → 5.0.0
-
- (:gh:`5621`, :gh:`5632`)
-
-- Fixes test and documentation issues (:gh:`5612`, :gh:`5617`,
- :gh:`5631`).
-
-
-.. _release-2.6.2:
-
-Scrapy 2.6.2 (2022-07-25)
--------------------------
-
-**Security bug fix:**
-
-- When :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
- processes a request with :reqmeta:`proxy` metadata, and that
- :reqmeta:`proxy` metadata includes proxy credentials,
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` sets
- the ``Proxy-Authorization`` header, but only if that header is not already
- set.
-
- There are third-party proxy-rotation downloader middlewares that set
- different :reqmeta:`proxy` metadata every time they process a request.
-
- Because of request retries and redirects, the same request can be processed
- by downloader middlewares more than once, including both
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and
- any third-party proxy-rotation downloader middleware.
-
- These third-party proxy-rotation downloader middlewares could change the
- :reqmeta:`proxy` metadata of a request to a new value, but fail to remove
- the ``Proxy-Authorization`` header from the previous value of the
- :reqmeta:`proxy` metadata, causing the credentials of one proxy to be sent
- to a different proxy.
-
- To prevent the unintended leaking of proxy credentials, the behavior of
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` is now
- as follows when processing a request:
-
- - If the request being processed defines :reqmeta:`proxy` metadata that
- includes credentials, the ``Proxy-Authorization`` header is always
- updated to feature those credentials.
-
- - If the request being processed defines :reqmeta:`proxy` metadata
- without credentials, the ``Proxy-Authorization`` header is removed
- *unless* it was originally defined for the same proxy URL.
-
- To remove proxy credentials while keeping the same proxy URL, remove
- the ``Proxy-Authorization`` header.
-
- - If the request has no :reqmeta:`proxy` metadata, or that metadata is a
- falsy value (e.g. ``None``), the ``Proxy-Authorization`` header is
- removed.
-
- It is no longer possible to set a proxy URL through the
- :reqmeta:`proxy` metadata but set the credentials through the
- ``Proxy-Authorization`` header. Set proxy credentials through the
- :reqmeta:`proxy` metadata instead.
-
-Also fixes the following regressions introduced in 2.6.0:
-
-- :class:`~scrapy.crawler.CrawlerProcess` supports again crawling multiple
- spiders (:gh:`5435`, :gh:`5436`)
-
-- Installing a Twisted reactor before Scrapy does (e.g. importing
- :mod:`twisted.internet.reactor` somewhere at the module level) no longer
- prevents Scrapy from starting, as long as a different reactor is not
- specified in :setting:`TWISTED_REACTOR` (:gh:`5525`, :gh:`5528`)
-
-- Fixed an exception that was being logged after the spider finished under
- certain conditions (:gh:`5437`, :gh:`5440`)
-
-- The ``--output``/``-o`` command-line parameter supports again a value
- starting with a hyphen (:gh:`5444`, :gh:`5445`)
-
-- The ``scrapy parse -h`` command no longer throws an error (:gh:`5481`,
- :gh:`5482`)
-
-
-.. _release-2.6.1:
-
-Scrapy 2.6.1 (2022-03-01)
--------------------------
-
-Fixes a regression introduced in 2.6.0 that would unset the request method when
-following redirects.
-
-
-.. _release-2.6.0:
-
-Scrapy 2.6.0 (2022-03-01)
--------------------------
-
-Highlights:
-
-* :ref:`Security fixes for cookie handling <2.6-security-fixes>`
-
-* Python 3.10 support
-
-* :ref:`asyncio support <using-asyncio>` is no longer considered
- experimental, and works out-of-the-box on Windows regardless of your Python
- version
-
-* Feed exports now support :class:`pathlib.Path` output paths and per-feed
- :ref:`item filtering <item-filter>` and
- :ref:`post-processing <post-processing>`
-
-.. _2.6-security-fixes:
-
-Security bug fixes
-~~~~~~~~~~~~~~~~~~
-
-- When a :class:`~scrapy.Request` object with cookies defined gets a
- redirect response causing a new :class:`~scrapy.Request` object to be
- scheduled, the cookies defined in the original
- :class:`~scrapy.Request` object are no longer copied into the new
- :class:`~scrapy.Request` object.
-
- If you manually set the ``Cookie`` header on a
- :class:`~scrapy.Request` object and the domain name of the redirect
- URL is not an exact match for the domain of the URL of the original
- :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
- from the new :class:`~scrapy.Request` object.
-
- The old behavior could be exploited by an attacker to gain access to your
- cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
- information.
-
- .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8
-
- .. note:: It is still possible to enable the sharing of cookies between
- different domains with a shared domain suffix (e.g.
- ``example.com`` and any subdomain) by defining the shared domain
- suffix (e.g. ``example.com``) as the cookie domain when defining
- your cookies. See the documentation of the
- :class:`~scrapy.Request` class for more information.
-
-- When the domain of a cookie, either received in the ``Set-Cookie`` header
- of a response or defined in a :class:`~scrapy.Request` object, is set
- to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
- ignored unless the cookie domain is the same as the request domain.
-
- The old behavior could be exploited by an attacker to inject cookies from a
- controlled domain into your cookiejar that could be sent to other domains
- not controlled by the attacker. Please, see the `mfjm-vh54-3f96 security
- advisory`_ for more information.
-
- .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96
-
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-- The h2_ dependency is now optional, only needed to
- :ref:`enable HTTP/2 support <twisted-http2-handler>`. (:gh:`5113`)
-
- .. _h2: https://pypi.org/project/h2/
-
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- The ``formdata`` parameter of :class:`~scrapy.FormRequest`, if specified
- for a non-POST request, now overrides the URL query string, instead of
- being appended to it. (:gh:`2919`, :gh:`3579`)
-
-- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting, now
- the return value of that function, and not the ``params`` input parameter,
- will determine the feed URI parameters, unless that return value is
- ``None``. (:gh:`4962`, :gh:`4966`)
-
-- In :class:`scrapy.core.engine.ExecutionEngine`, methods
- :meth:`~scrapy.core.engine.ExecutionEngine.crawl`,
- :meth:`~scrapy.core.engine.ExecutionEngine.download`,
- :meth:`~scrapy.core.engine.ExecutionEngine.schedule`,
- and :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
- now raise :exc:`RuntimeError` if called before
- :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`. (:gh:`5090`)
-
- These methods used to assume that
- :attr:`ExecutionEngine.slot <scrapy.core.engine.ExecutionEngine.slot>` had
- been defined by a prior call to
- :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`, so they were
- raising :exc:`AttributeError` instead.
-
-- If the API of the configured :ref:`scheduler <topics-scheduler>` does not
- meet expectations, :exc:`TypeError` is now raised at startup time. Before,
- other exceptions would be raised at run time. (:gh:`3559`)
-
-- The ``_encoding`` field of serialized :class:`~scrapy.Request` objects
- is now named ``encoding``, in line with all other fields (:gh:`5130`)
-
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- ``scrapy.http.TextResponse.body_as_unicode``, deprecated in Scrapy 2.2, has
- now been removed. (:gh:`5393`)
-
-- ``scrapy.item.BaseItem``, deprecated in Scrapy 2.2, has now been removed.
- (:gh:`5398`)
-
-- ``scrapy.item.DictItem``, deprecated in Scrapy 1.8, has now been removed.
- (:gh:`5398`)
-
-- ``scrapy.Spider.make_requests_from_url``, deprecated in Scrapy 1.4, has now
- been removed. (:gh:`4178`, :gh:`4356`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
- returning ``None`` or modifying the ``params`` input parameter is now
- deprecated. Return a new dictionary instead. (:gh:`4962`, :gh:`4966`)
-
-- :mod:`scrapy.utils.reqser` is deprecated. (:gh:`5130`)
-
- - Instead of :func:`~scrapy.utils.reqser.request_to_dict`, use the new
- :meth:`.Request.to_dict` method.
-
- - Instead of :func:`~scrapy.utils.reqser.request_from_dict`, use the new
- :func:`scrapy.utils.request.request_from_dict` function.
-
-- In :mod:`scrapy.squeues`, the following queue classes are deprecated:
- :class:`~scrapy.squeues.PickleFifoDiskQueueNonRequest`,
- :class:`~scrapy.squeues.PickleLifoDiskQueueNonRequest`,
- :class:`~scrapy.squeues.MarshalFifoDiskQueueNonRequest`,
- and :class:`~scrapy.squeues.MarshalLifoDiskQueueNonRequest`. You should
- instead use:
- :class:`~scrapy.squeues.PickleFifoDiskQueue`,
- :class:`~scrapy.squeues.PickleLifoDiskQueue`,
- :class:`~scrapy.squeues.MarshalFifoDiskQueue`,
- and :class:`~scrapy.squeues.MarshalLifoDiskQueue`. (:gh:`5117`)
-
-- Many aspects of :class:`scrapy.core.engine.ExecutionEngine` that come from
- a time when this class could handle multiple :class:`~scrapy.Spider`
- objects at a time have been deprecated. (:gh:`5090`)
-
- - The :meth:`~scrapy.core.engine.ExecutionEngine.has_capacity` method
- is deprecated.
-
- - The :meth:`~scrapy.core.engine.ExecutionEngine.schedule` method is
- deprecated, use :meth:`~scrapy.core.engine.ExecutionEngine.crawl` or
- :meth:`~scrapy.core.engine.ExecutionEngine.download` instead.
-
- - The :attr:`~scrapy.core.engine.ExecutionEngine.open_spiders` attribute
- is deprecated, use :attr:`~scrapy.core.engine.ExecutionEngine.spider`
- instead.
-
- - The ``spider`` parameter is deprecated for the following methods:
-
- - :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
-
- - :meth:`~scrapy.core.engine.ExecutionEngine.crawl`
-
- - :meth:`~scrapy.core.engine.ExecutionEngine.download`
-
- Instead, call :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`
- first to set the :class:`~scrapy.Spider` object.
-
-- :func:`scrapy.utils.response.response_httprepr` is now deprecated.
- (:gh:`4972`)
-
-
-New features
-~~~~~~~~~~~~
-
-- You can now use :ref:`item filtering <item-filter>` to control which items
- are exported to each output feed. (:gh:`4575`, :gh:`5178`,
- :gh:`5161`, :gh:`5203`)
-
-- You can now apply :ref:`post-processing <post-processing>` to feeds, and
- :ref:`built-in post-processing plugins <builtin-plugins>` are provided for
- output file compression. (:gh:`2174`, :gh:`5168`, :gh:`5190`)
-
-- The :setting:`FEEDS` setting now supports :class:`pathlib.Path` objects as
- keys. (:gh:`5383`, :gh:`5384`)
-
-- Enabling :ref:`asyncio <using-asyncio>` while using Windows and Python 3.8
- or later will automatically switch the asyncio event loop to one that
- allows Scrapy to work. See :ref:`asyncio-windows`. (:gh:`4976`,
- :gh:`5315`)
-
-- The :command:`genspider` command now supports a start URL instead of a
- domain name. (:gh:`4439`)
-
-- :mod:`scrapy.utils.defer` gained 2 new functions,
- :func:`~scrapy.utils.defer.deferred_to_future` and
- :func:`~scrapy.utils.defer.maybe_deferred_to_future`, to help :ref:`await
- on Deferreds when using the asyncio reactor <asyncio-await-dfd>`.
- (:gh:`5288`)
-
-- :ref:`Amazon S3 feed export storage <topics-feed-storage-s3>` gained
- support for `temporary security credentials`_
- (:setting:`AWS_SESSION_TOKEN`) and endpoint customization
- (:setting:`AWS_ENDPOINT_URL`). (:gh:`4998`, :gh:`5210`)
-
- .. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
-
-- New :setting:`LOG_FILE_APPEND` setting to allow truncating the log file.
- (:gh:`5279`)
-
-- :attr:`Request.cookies <scrapy.Request.cookies>` values that are
- :class:`bool`, :class:`float` or :class:`int` are cast to :class:`str`.
- (:gh:`5252`, :gh:`5253`)
-
-- You may now raise :exc:`~scrapy.exceptions.CloseSpider` from a handler of
- the :signal:`spider_idle` signal to customize the reason why the spider is
- stopping. (:gh:`5191`)
-
-- When using
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`, the
- proxy URL for non-HTTPS HTTP/1.1 requests no longer needs to include a URL
- scheme. (:gh:`4505`, :gh:`4649`)
-
-- All built-in queues now expose a ``peek`` method that returns the next
- queue object (like ``pop``) but does not remove the returned object from
- the queue. (:gh:`5112`)
-
- If the underlying queue does not support peeking (e.g. because you are not
- using ``queuelib`` 1.6.1 or later), the ``peek`` method raises
- :exc:`NotImplementedError`.
-
-- :class:`~scrapy.Request` and :class:`~scrapy.http.Response` now have
- an ``attributes`` attribute that makes subclassing easier. For
- :class:`~scrapy.Request`, it also allows subclasses to work with
- :func:`scrapy.utils.request.request_from_dict`. (:gh:`1877`,
- :gh:`5130`, :gh:`5218`)
-
-- The :meth:`~scrapy.core.scheduler.BaseScheduler.open` and
- :meth:`~scrapy.core.scheduler.BaseScheduler.close` methods of the
- :ref:`scheduler <topics-scheduler>` are now optional. (:gh:`3559`)
-
-- HTTP/1.1 :exc:`~scrapy.core.downloader.handlers.http11.TunnelError`
- exceptions now only truncate response bodies longer than 1000 characters,
- instead of those longer than 32 characters, making it easier to debug such
- errors. (:gh:`4881`, :gh:`5007`)
-
-- :class:`~scrapy.loader.ItemLoader` now supports non-text responses.
- (:gh:`5145`, :gh:`5269`)
-
-
-Bug fixes
-~~~~~~~~~
-
-- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings
- are no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`.
- (:gh:`4485`, :gh:`5352`)
-
-- Removed a module-level Twisted reactor import that could prevent
- :ref:`using the asyncio reactor <using-asyncio>`. (:gh:`5357`)
-
-- The :command:`startproject` command works with existing folders again.
- (:gh:`4665`, :gh:`4676`)
-
-- The :setting:`FEED_URI_PARAMS` setting now behaves as documented.
- (:gh:`4962`, :gh:`4966`)
-
-- :attr:`Request.cb_kwargs <scrapy.Request.cb_kwargs>` once again allows the
- ``callback`` keyword. (:gh:`5237`, :gh:`5251`, :gh:`5264`)
-
-- Made :func:`scrapy.utils.response.open_in_browser` support more complex
- HTML. (:gh:`5319`, :gh:`5320`)
-
-- Fixed :attr:`CSVFeedSpider.quotechar
- <scrapy.spiders.CSVFeedSpider.quotechar>` being interpreted as the CSV file
- encoding. (:gh:`5391`, :gh:`5394`)
-
-- Added missing setuptools_ to the list of dependencies. (:gh:`5122`)
-
- .. _setuptools: https://pypi.org/project/setuptools/
-
-- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- now also works as expected with links that have comma-separated ``rel``
- attribute values including ``nofollow``. (:gh:`5225`)
-
-- Fixed a :exc:`TypeError` that could be raised during :ref:`feed export
- <topics-feed-exports>` parameter parsing. (:gh:`5359`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-- :ref:`asyncio support <using-asyncio>` is no longer considered
- experimental. (:gh:`5332`)
-
-- Included :ref:`Windows-specific help for asyncio usage <asyncio-windows>`.
- (:gh:`4976`, :gh:`5315`)
-
-- Rewrote :ref:`topics-headless-browsing` with up-to-date best practices.
- (:gh:`4484`, :gh:`4613`)
-
-- Documented :ref:`local file naming in media pipelines
- <topics-file-naming>`. (:gh:`5069`, :gh:`5152`)
-
-- :ref:`faq` now covers spider file name collision issues. (:gh:`2680`,
- :gh:`3669`)
-
-- Provided better context and instructions to disable the
- :setting:`URLLENGTH_LIMIT` setting. (:gh:`5135`, :gh:`5250`)
-
-- Documented that Reppy parser does not support Python 3.9+.
- (:gh:`5226`, :gh:`5231`)
-
-- Documented :ref:`the scheduler component <topics-scheduler>`.
- (:gh:`3537`, :gh:`3559`)
-
-- Documented the method used by :ref:`media pipelines
- <topics-media-pipeline>` to :ref:`determine if a file has expired
- <file-expiration>`. (:gh:`5120`, :gh:`5254`)
-
-- :ref:`run-multiple-spiders` now features
- :func:`scrapy.utils.project.get_project_settings` usage. (:gh:`5070`)
-
-- :ref:`run-multiple-spiders` now covers what happens when you define
- different per-spider values for some settings that cannot differ at run
- time. (:gh:`4485`, :gh:`5352`)
-
-- Extended the documentation of the
- :class:`~scrapy.extensions.statsmailer.StatsMailer` extension.
- (:gh:`5199`, :gh:`5217`)
-
-- Added :setting:`JOBDIR` to :ref:`topics-settings`. (:gh:`5173`,
- :gh:`5224`)
-
-- Documented :attr:`Spider.attribute <scrapy.Spider.attribute>`.
- (:gh:`5174`, :gh:`5244`)
-
-- Documented :attr:`TextResponse.urljoin <scrapy.http.TextResponse.urljoin>`.
- (:gh:`1582`)
-
-- Added the ``body_length`` parameter to the documented signature of the
- :signal:`headers_received` signal. (:gh:`5270`)
-
-- Clarified :meth:`SelectorList.get <scrapy.selector.SelectorList.get>` usage
- in the :ref:`tutorial <intro-tutorial>`. (:gh:`5256`)
-
-- The documentation now features the shortest import path of classes with
- multiple import paths. (:gh:`2733`, :gh:`5099`)
-
-- ``quotes.toscrape.com`` references now use HTTPS instead of HTTP.
- (:gh:`5395`, :gh:`5396`)
-
-- Added a link to `our Discord server <https://discord.com/invite/mv3yErfpvq>`_
- to :ref:`getting-help`. (:gh:`5421`, :gh:`5422`)
-
-- The pronunciation of the project name is now :ref:`officially
- <intro-overview>` /ˈskreɪpaɪ/. (:gh:`5280`, :gh:`5281`)
-
-- Added the Scrapy logo to the README. (:gh:`5255`, :gh:`5258`)
-
-- Fixed issues and implemented minor improvements. (:gh:`3155`,
- :gh:`4335`, :gh:`5074`, :gh:`5098`, :gh:`5134`, :gh:`5180`,
- :gh:`5194`, :gh:`5239`, :gh:`5266`, :gh:`5271`, :gh:`5273`,
- :gh:`5274`, :gh:`5276`, :gh:`5347`, :gh:`5356`, :gh:`5414`,
- :gh:`5415`, :gh:`5416`, :gh:`5419`, :gh:`5420`)
-
-
-Quality Assurance
-~~~~~~~~~~~~~~~~~
-
-- Added support for Python 3.10. (:gh:`5212`, :gh:`5221`,
- :gh:`5265`)
-
-- Significantly reduced memory usage by
- :func:`scrapy.utils.response.response_httprepr`, used by the
- :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` downloader
- middleware, which is enabled by default. (:gh:`4964`, :gh:`4972`)
-
-- Removed uses of the deprecated :mod:`optparse` module. (:gh:`5366`,
- :gh:`5374`)
-
-- Extended typing hints. (:gh:`5077`, :gh:`5090`, :gh:`5100`,
- :gh:`5108`, :gh:`5171`, :gh:`5215`, :gh:`5334`)
-
-- Improved tests, fixed CI issues, removed unused code. (:gh:`5094`,
- :gh:`5157`, :gh:`5162`, :gh:`5198`, :gh:`5207`, :gh:`5208`,
- :gh:`5229`, :gh:`5298`, :gh:`5299`, :gh:`5310`, :gh:`5316`,
- :gh:`5333`, :gh:`5388`, :gh:`5389`, :gh:`5400`, :gh:`5401`,
- :gh:`5404`, :gh:`5405`, :gh:`5407`, :gh:`5410`, :gh:`5412`,
- :gh:`5425`, :gh:`5427`)
-
-- Implemented improvements for contributors. (:gh:`5080`, :gh:`5082`,
- :gh:`5177`, :gh:`5200`)
-
-- Implemented cleanups. (:gh:`5095`, :gh:`5106`, :gh:`5209`,
- :gh:`5228`, :gh:`5235`, :gh:`5245`, :gh:`5246`, :gh:`5292`,
- :gh:`5314`, :gh:`5322`)
-
-
-.. _release-2.5.1:
-
-Scrapy 2.5.1 (2021-10-05)
--------------------------
-
-* **Security bug fix:**
-
- If you use
- :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
- (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP
- authentication, any request exposes your credentials to the request target.
-
- To prevent unintended exposure of authentication credentials to unintended
- domains, you must now additionally set a new, additional spider attribute,
- ``http_auth_domain``, and point it to the specific domain to which the
- authentication credentials must be sent.
-
- If the ``http_auth_domain`` spider attribute is not set, the domain of the
- first request will be considered the HTTP authentication target, and
- authentication credentials will only be sent in requests targeting that
- domain.
-
- If you need to send the same HTTP authentication credentials to multiple
- domains, you can use :func:`w3lib.http.basic_auth_header` instead to
- set the value of the ``Authorization`` header of your requests.
-
- If you *really* want your spider to send the same HTTP authentication
- credentials to any domain, set the ``http_auth_domain`` spider attribute
- to ``None``.
-
- Finally, if you are a user of `scrapy-splash`_, know that this version of
- Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will
- need to upgrade scrapy-splash to a greater version for it to continue to
- work.
-
-
-.. _release-2.5.0:
-
-Scrapy 2.5.0 (2021-04-06)
--------------------------
-
-Highlights:
-
-- Official Python 3.9 support
-
-- Experimental :ref:`HTTP/2 support <twisted-http2-handler>`
-
-- New :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function
- to retry requests from spider callbacks
-
-- New :class:`~scrapy.signals.headers_received` signal that allows stopping
- downloads early
-
-- New :class:`Response.protocol <scrapy.http.Response.protocol>` attribute
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-- Removed all code that :ref:`was deprecated in 1.7.0 <1.7-deprecations>` and
- had not :ref:`already been removed in 2.4.0 <2.4-deprecation-removals>`.
- (:gh:`4901`)
-
-- Removed support for the ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
- variable, :ref:`deprecated in 1.8.0 <1.8-deprecations>`. (:gh:`4912`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-- The :mod:`scrapy.utils.py36` module is now deprecated in favor of
- :mod:`scrapy.utils.asyncgen`. (:gh:`4900`)
-
-
-New features
-~~~~~~~~~~~~
-
-- Experimental :ref:`HTTP/2 support <twisted-http2-handler>` through a new download handler
- that can be assigned to the ``https`` protocol in the
- :setting:`DOWNLOAD_HANDLERS` setting.
- (:gh:`1854`, :gh:`4769`, :gh:`5058`, :gh:`5059`, :gh:`5066`)
-
-- The new :func:`scrapy.downloadermiddlewares.retry.get_retry_request`
- function may be used from spider callbacks or middlewares to handle the
- retrying of a request beyond the scenarios that
- :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` supports.
- (:gh:`3590`, :gh:`3685`, :gh:`4902`)
-
-- The new :class:`~scrapy.signals.headers_received` signal gives early access
- to response headers and allows :ref:`stopping downloads
- <topics-stop-response-download>`.
- (:gh:`1772`, :gh:`4897`)
-
-- The new :attr:`Response.protocol <scrapy.http.Response.protocol>`
- attribute gives access to the string that identifies the protocol used to
- download a response. (:gh:`4878`)
-
-- :ref:`Stats <topics-stats>` now include the following entries that indicate
- the number of successes and failures in storing
- :ref:`feeds <topics-feed-exports>`::
-
- feedexport/success_count/<storage type>
- feedexport/failed_count/<storage type>
-
- Where ``<storage type>`` is the feed storage backend class name, such as
- :class:`~scrapy.extensions.feedexport.FileFeedStorage` or
- :class:`~scrapy.extensions.feedexport.FTPFeedStorage`.
-
- (:gh:`3947`, :gh:`4850`)
-
-- The :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` spider
- middleware now logs ignored URLs with ``INFO`` :ref:`logging level
- <levels>` instead of ``DEBUG``, and it now includes the following entry
- into :ref:`stats <topics-stats>` to keep track of the number of ignored
- URLs::
-
- urllength/request_ignored_count
-
- (:gh:`5036`)
-
-- The
- :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
- downloader middleware now logs the number of decompressed responses and the
- total count of resulting bytes::
-
- httpcompression/response_bytes
- httpcompression/response_count
-
- (:gh:`4797`, :gh:`4799`)
-
-
-Bug fixes
-~~~~~~~~~
-
-- Fixed installation on PyPy installing PyDispatcher in addition to
- PyPyDispatcher, which could prevent Scrapy from working depending on which
- package got imported. (:gh:`4710`, :gh:`4814`)
-
-- When inspecting a callback to check if it is a generator that also returns
- a value, an exception is no longer raised if the callback has a docstring
- with lower indentation than the following code.
- (:gh:`4477`, :gh:`4935`)
-
-- The `Content-Length <https://datatracker.ietf.org/doc/html/rfc2616#section-14.13>`_
- header is no longer omitted from responses when using the default, HTTP/1.1
- download handler (see :setting:`DOWNLOAD_HANDLERS`).
- (:gh:`5009`, :gh:`5034`, :gh:`5045`, :gh:`5057`, :gh:`5062`)
-
-- Setting the :reqmeta:`handle_httpstatus_all` request meta key to ``False``
- now has the same effect as not setting it at all, instead of having the
- same effect as setting it to ``True``.
- (:gh:`3851`, :gh:`4694`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-- Added instructions to :ref:`install Scrapy in Windows using pip
- <intro-install-windows>`.
- (:gh:`4715`, :gh:`4736`)
-
-- Logging documentation now includes :ref:`additional ways to filter logs
- <topics-logging-advanced-customization>`.
- (:gh:`4216`, :gh:`4257`, :gh:`4965`)
-
-- Covered how to deal with long lists of allowed domains in the :ref:`FAQ
- <faq>`. (:gh:`2263`, :gh:`3667`)
-
-- Covered scrapy-bench_ in :ref:`benchmarking`.
- (:gh:`4996`, :gh:`5016`)
-
-- Clarified that one :ref:`extension <topics-extensions>` instance is created
- per crawler.
- (:gh:`5014`)
-
-- Fixed some errors in examples.
- (:gh:`4829`, :gh:`4830`, :gh:`4907`, :gh:`4909`,
- :gh:`5008`)
-
-- Fixed some external links, typos, and so on.
- (:gh:`4892`, :gh:`4899`, :gh:`4936`, :gh:`4942`, :gh:`5005`,
- :gh:`5063`)
-
-- The :ref:`list of Request.meta keys <topics-request-meta>` is now sorted
- alphabetically.
- (:gh:`5061`, :gh:`5065`)
-
-- Updated references to Scrapinghub, which is now called Zyte.
- (:gh:`4973`, :gh:`5072`)
-
-- Added a mention to contributors in the README. (:gh:`4956`)
-
-- Reduced the top margin of lists. (:gh:`4974`)
-
-
-Quality Assurance
-~~~~~~~~~~~~~~~~~
-
-- Made Python 3.9 support official (:gh:`4757`, :gh:`4759`)
-
-- Extended typing hints (:gh:`4895`)
-
-- Fixed deprecated uses of the Twisted API.
- (:gh:`4940`, :gh:`4950`, :gh:`5073`)
-
-- Made our tests run with the new pip resolver.
- (:gh:`4710`, :gh:`4814`)
-
-- Added tests to ensure that :ref:`coroutine support <coroutine-support>`
- is tested. (:gh:`4987`)
-
-- Migrated from Travis CI to GitHub Actions. (:gh:`4924`)
-
-- Fixed CI issues.
- (:gh:`4986`, :gh:`5020`, :gh:`5022`, :gh:`5027`, :gh:`5052`,
- :gh:`5053`)
-
-- Implemented code refactorings, style fixes and cleanups.
- (:gh:`4911`, :gh:`4982`, :gh:`5001`, :gh:`5002`, :gh:`5076`)
-
-
-.. _release-2.4.1:
-
-Scrapy 2.4.1 (2020-11-17)
--------------------------
-
-- Fixed :ref:`feed exports <topics-feed-exports>` overwrite support (:gh:`4845`, :gh:`4857`, :gh:`4859`)
-
-- Fixed the AsyncIO event loop handling, which could make code hang
- (:gh:`4855`, :gh:`4872`)
-
-- Fixed the IPv6-capable DNS resolver
- :class:`~scrapy.resolver.CachingHostnameResolver` for download handlers
- that call
- :meth:`reactor.resolve <twisted.internet.interfaces.IReactorCore.resolve>`
- (:gh:`4802`, :gh:`4803`)
-
-- Fixed the output of the :command:`genspider` command showing placeholders
- instead of the import path of the generated spider module (:gh:`4874`)
-
-- Migrated Windows CI from Azure Pipelines to GitHub Actions (:gh:`4869`,
- :gh:`4876`)
-
-
-.. _release-2.4.0:
-
-Scrapy 2.4.0 (2020-10-11)
--------------------------
-
-Highlights:
-
-* Python 3.5 support has been dropped.
-
-* The ``file_path`` method of :ref:`media pipelines <topics-media-pipeline>`
- can now access the source :ref:`item <topics-items>`.
-
- This allows you to set a download file path based on item data.
-
-* The new ``item_export_kwargs`` key of the :setting:`FEEDS` setting allows
- to define keyword parameters to pass to :ref:`item exporter classes
- <topics-exporters>`
-
-* You can now choose whether :ref:`feed exports <topics-feed-exports>`
- overwrite or append to the output file.
-
- For example, when using the :command:`crawl` or :command:`runspider`
- commands, you can use the ``-O`` option instead of ``-o`` to overwrite the
- output file.
-
-* Zstd-compressed responses are now supported if zstandard_ is installed.
-
-* In settings, where the import path of a class is required, it is now
- possible to pass a class object instead.
-
-Modified requirements
-~~~~~~~~~~~~~~~~~~~~~
-
-* Python 3.6 or greater is now required; support for Python 3.5 has been
- dropped
-
- As a result:
-
- - When using PyPy, PyPy 7.2.0 or greater :ref:`is now required
- <faq-python-versions>`
-
- - For Amazon S3 storage support in :ref:`feed exports
- <topics-feed-storage-s3>` or :ref:`media pipelines
- <media-pipelines-s3>`, botocore_ 1.4.87 or greater is now required
-
- - To use the :ref:`images pipeline <images-pipeline>`, Pillow_ 4.0.0 or
- greater is now required
-
- (:gh:`4718`, :gh:`4732`, :gh:`4733`, :gh:`4742`, :gh:`4743`,
- :gh:`4764`)
-
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` once again
- discards cookies defined in :attr:`.Request.headers`.
-
- We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it
- was reported that the current implementation could break existing code.
-
- If you need to set cookies for a request, use the :class:`Request.cookies
- <scrapy.Request>` parameter.
-
- A future version of Scrapy will include a new, better implementation of the
- reverted bug fix.
-
- (:gh:`4717`, :gh:`4823`)
-
-
-.. _2.4-deprecation-removals:
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-* :class:`scrapy.extensions.feedexport.S3FeedStorage` no longer reads the
- values of ``access_key`` and ``secret_key`` from the running project
- settings when they are not passed to its ``__init__`` method; you must
- either pass those parameters to its ``__init__`` method or use
- :class:`S3FeedStorage.from_crawler
- <scrapy.extensions.feedexport.S3FeedStorage.from_crawler>`
- (:gh:`4356`, :gh:`4411`, :gh:`4688`)
-
-* :attr:`Rule.process_request <scrapy.spiders.crawl.Rule.process_request>`
- no longer admits callables which expect a single ``request`` parameter,
- rather than both ``request`` and ``response`` (:gh:`4818`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-* In custom :ref:`media pipelines <topics-media-pipeline>`, signatures that
- do not accept a keyword-only ``item`` parameter in any of the methods that
- :ref:`now support this parameter <media-pipeline-item-parameter>` are now
- deprecated (:gh:`4628`, :gh:`4686`)
-
-* In custom :ref:`feed storage backend classes <topics-feed-storage>`,
- ``__init__`` method signatures that do not accept a keyword-only
- ``feed_options`` parameter are now deprecated (:gh:`547`, :gh:`716`,
- :gh:`4512`)
-
-* The :class:`scrapy.utils.python.WeakKeyCache` class is now deprecated
- (:gh:`4684`, :gh:`4701`)
-
-* The :func:`scrapy.utils.boto.is_botocore` function is now deprecated, use
- :func:`scrapy.utils.boto.is_botocore_available` instead (:gh:`4734`,
- :gh:`4776`)
-
-
-New features
-~~~~~~~~~~~~
-
-.. _media-pipeline-item-parameter:
-
-* The following methods of :ref:`media pipelines <topics-media-pipeline>` now
- accept an ``item`` keyword-only parameter containing the source
- :ref:`item <topics-items>`:
-
- - In :class:`scrapy.pipelines.files.FilesPipeline`:
-
- - :meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded`
-
- - :meth:`~scrapy.pipelines.files.FilesPipeline.file_path`
-
- - :meth:`~scrapy.pipelines.files.FilesPipeline.media_downloaded`
-
- - :meth:`~scrapy.pipelines.files.FilesPipeline.media_to_download`
-
- - In :class:`scrapy.pipelines.images.ImagesPipeline`:
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.file_downloaded`
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.file_path`
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.get_images`
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.image_downloaded`
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.media_downloaded`
-
- - :meth:`~scrapy.pipelines.images.ImagesPipeline.media_to_download`
-
- (:gh:`4628`, :gh:`4686`)
-
-* The new ``item_export_kwargs`` key of the :setting:`FEEDS` setting allows
- to define keyword parameters to pass to :ref:`item exporter classes
- <topics-exporters>` (:gh:`4606`, :gh:`4768`)
-
-* :ref:`Feed exports <topics-feed-exports>` gained overwrite support:
-
- * When using the :command:`crawl` or :command:`runspider` commands, you
- can use the ``-O`` option instead of ``-o`` to overwrite the output
- file
-
- * You can use the ``overwrite`` key in the :setting:`FEEDS` setting to
- configure whether to overwrite the output file (``True``) or append to
- its content (``False``)
-
- * The ``__init__`` and ``from_crawler`` methods of :ref:`feed storage
- backend classes <topics-feed-storage>` now receive a new keyword-only
- parameter, ``feed_options``, which is a dictionary of :ref:`feed
- options <feed-options>`
-
- (:gh:`547`, :gh:`716`, :gh:`4512`)
-
-* Zstd-compressed responses are now supported if zstandard_ is installed
- (:gh:`4831`)
-
-* In settings, where the import path of a class is required, it is now
- possible to pass a class object instead (:gh:`3870`, :gh:`3873`).
-
- This includes also settings where only part of its value is made of an
- import path, such as :setting:`DOWNLOADER_MIDDLEWARES` or
- :setting:`DOWNLOAD_HANDLERS`.
-
-* :ref:`Downloader middlewares <topics-downloader-middleware>` can now
- override :class:`response.request <scrapy.http.Response.request>`.
-
- If a :ref:`downloader middleware <topics-downloader-middleware>` returns
- a :class:`~scrapy.http.Response` object from
- :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`
- or
- :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
- with a custom :class:`~scrapy.Request` object assigned to
- :class:`response.request <scrapy.http.Response.request>`:
-
- - The response is handled by the callback of that custom
- :class:`~scrapy.Request` object, instead of being handled by the
- callback of the original :class:`~scrapy.Request` object
-
- - That custom :class:`~scrapy.Request` object is now sent as the
- ``request`` argument to the :signal:`response_received` signal, instead
- of the original :class:`~scrapy.Request` object
-
- (:gh:`4529`, :gh:`4632`)
-
-* When using the :ref:`FTP feed storage backend <topics-feed-storage-ftp>`:
-
- - It is now possible to set the new ``overwrite`` :ref:`feed option
- <feed-options>` to ``False`` to append to an existing file instead of
- overwriting it
-
- - The FTP password can now be omitted if it is not necessary
-
- (:gh:`547`, :gh:`716`, :gh:`4512`)
-
-* The ``__init__`` method of :class:`~scrapy.exporters.CsvItemExporter` now
- supports an ``errors`` parameter to indicate how to handle encoding errors
- (:gh:`4755`)
-
-* When :ref:`using asyncio <using-asyncio>`, it is now possible to
- :ref:`set a custom asyncio loop <using-custom-loops>` (:gh:`4306`,
- :gh:`4414`)
-
-* Serialized requests (see :ref:`topics-jobs`) now support callbacks that are
- spider methods that delegate on other callable (:gh:`4756`)
-
-* When a response is larger than :setting:`DOWNLOAD_MAXSIZE`, the logged
- message is now a warning, instead of an error (:gh:`3874`,
- :gh:`3886`, :gh:`4752`)
-
-
-Bug fixes
-~~~~~~~~~
-
-* The :command:`genspider` command no longer overwrites existing files
- unless the ``--force`` option is used (:gh:`4561`, :gh:`4616`,
- :gh:`4623`)
-
-* Cookies with an empty value are no longer considered invalid cookies
- (:gh:`4772`)
-
-* The :command:`runspider` command now supports files with the ``.pyw`` file
- extension (:gh:`4643`, :gh:`4646`)
-
-* The :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
- middleware now simply ignores unsupported proxy values (:gh:`3331`,
- :gh:`4778`)
-
-* Checks for generator callbacks with a ``return`` statement no longer warn
- about ``return`` statements in nested functions (:gh:`4720`,
- :gh:`4721`)
-
-* The system file mode creation mask no longer affects the permissions of
- files generated using the :command:`startproject` command (:gh:`4722`)
-
-* ``scrapy.utils.iterators.xmliter`` now supports namespaced node names
- (:gh:`861`, :gh:`4746`)
-
-* :class:`~scrapy.Request` objects can now have ``about:`` URLs, which can
- work when using a headless browser (:gh:`4835`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* The :setting:`FEED_URI_PARAMS` setting is now documented (:gh:`4671`,
- :gh:`4724`)
-
-* Improved the documentation of
- :ref:`link extractors <topics-link-extractors>` with an usage example from
- a spider callback and reference documentation for the
- :class:`~scrapy.link.Link` class (:gh:`4751`, :gh:`4775`)
-
-* Clarified the impact of :setting:`CONCURRENT_REQUESTS` when using the
- :class:`~scrapy.extensions.closespider.CloseSpider` extension
- (:gh:`4836`)
-
-* Removed references to Python 2’s ``unicode`` type (:gh:`4547`,
- :gh:`4703`)
-
-* We now have an :ref:`official deprecation policy <deprecation-policy>`
- (:gh:`4705`)
-
-* Our :ref:`documentation policies <documentation-policies>` now cover usage
- of Sphinx’s :rst:dir:`versionadded` and :rst:dir:`versionchanged`
- directives, and we have removed usages referencing Scrapy 1.4.0 and earlier
- versions (:gh:`3971`, :gh:`4310`)
-
-* Other documentation cleanups (:gh:`4090`, :gh:`4782`, :gh:`4800`,
- :gh:`4801`, :gh:`4809`, :gh:`4816`, :gh:`4825`)
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-* Extended typing hints (:gh:`4243`, :gh:`4691`)
-
-* Added tests for the :command:`check` command (:gh:`4663`)
-
-* Fixed test failures on Debian (:gh:`4726`, :gh:`4727`, :gh:`4735`)
-
-* Improved Windows test coverage (:gh:`4723`)
-
-* Switched to :ref:`formatted string literals <f-strings>` where possible
- (:gh:`4307`, :gh:`4324`, :gh:`4672`)
-
-* Modernized :func:`super` usage (:gh:`4707`)
-
-* Other code and test cleanups (:gh:`1790`, :gh:`3288`, :gh:`4165`,
- :gh:`4564`, :gh:`4651`, :gh:`4714`, :gh:`4738`, :gh:`4745`,
- :gh:`4747`, :gh:`4761`, :gh:`4765`, :gh:`4804`, :gh:`4817`,
- :gh:`4820`, :gh:`4822`, :gh:`4839`)
-
-
-.. _release-2.3.0:
-
-Scrapy 2.3.0 (2020-08-04)
--------------------------
-
-Highlights:
-
-* :ref:`Feed exports <topics-feed-exports>` now support :ref:`Google Cloud
- Storage <topics-feed-storage-gcs>` as a storage backend
-
-* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver
- output items in batches of up to the specified number of items.
-
- It also serves as a workaround for :ref:`delayed file delivery
- <delayed-file-delivery>`, which causes Scrapy to only start item delivery
- after the crawl has finished when using certain storage backends
- (:ref:`S3 <topics-feed-storage-s3>`, :ref:`FTP <topics-feed-storage-ftp>`,
- and now :ref:`GCS <topics-feed-storage-gcs>`).
-
-* The base implementation of :ref:`item loaders <topics-loaders>` has been
- moved into a separate library, :doc:`itemloaders <itemloaders:index>`,
- allowing usage from outside Scrapy and a separate release schedule
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-* Removed the following classes and their parent modules from
- ``scrapy.linkextractors``:
-
- * ``htmlparser.HtmlParserLinkExtractor``
- * ``regex.RegexLinkExtractor``
- * ``sgml.BaseSgmlLinkExtractor``
- * ``sgml.SgmlLinkExtractor``
-
- Use
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- instead (:gh:`4356`, :gh:`4679`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated
- (:gh:`4683`)
-
-
-New features
-~~~~~~~~~~~~
-
-* :ref:`Feed exports <topics-feed-exports>` support :ref:`Google Cloud
- Storage <topics-feed-storage-gcs>` (:gh:`685`, :gh:`3608`)
-
-* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries
- (:gh:`4250`, :gh:`4434`)
-
-* The :command:`parse` command now allows specifying an output file
- (:gh:`4317`, :gh:`4377`)
-
-* :meth:`.Request.from_curl` and
- :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
- ``--data-raw`` (:gh:`4612`)
-
-* A ``parse`` callback may now be used in built-in spider subclasses, such
- as :class:`~scrapy.spiders.CrawlSpider` (:gh:`712`, :gh:`732`,
- :gh:`781`, :gh:`4254` )
-
-
-Bug fixes
-~~~~~~~~~
-
-* Fixed the :ref:`CSV exporting <topics-feed-format-csv>` of
- :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
- <attrs-items>` (:gh:`4667`, :gh:`4668`)
-
-* :meth:`.Request.from_curl` and
- :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
- method to ``POST`` when a request body is specified and no request method
- is specified (:gh:`4612`)
-
-* The processing of ANSI escape sequences in enabled in Windows 10.0.14393
- and later, where it is required for colored output (:gh:`4393`,
- :gh:`4403`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* Updated the `OpenSSL cipher list format`_ link in the documentation about
- the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:gh:`4653`)
-
-* Simplified the code example in :ref:`topics-loaders-dataclass`
- (:gh:`4652`)
-
-.. _OpenSSL cipher list format: https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-* The base implementation of :ref:`item loaders <topics-loaders>` has been
- moved into :doc:`itemloaders <itemloaders:index>` (:gh:`4005`,
- :gh:`4516`)
-
-* Fixed a silenced error in some scheduler tests (:gh:`4644`,
- :gh:`4645`)
-
-* Renewed the localhost certificate used for SSL tests (:gh:`4650`)
-
-* Removed cookie-handling code specific to Python 2 (:gh:`4682`)
-
-* Stopped using Python 2 unicode literal syntax (:gh:`4704`)
-
-* Stopped using a backlash for line continuation (:gh:`4673`)
-
-* Removed unneeded entries from the MyPy exception list (:gh:`4690`)
-
-* Automated tests now pass on Windows as part of our continuous integration
- system (:gh:`4458`)
-
-* Automated tests now pass on the latest PyPy version for supported Python
- versions in our continuous integration system (:gh:`4504`)
-
-
-.. _release-2.2.1:
-
-Scrapy 2.2.1 (2020-07-17)
--------------------------
-
-* The :command:`startproject` command no longer makes unintended changes to
- the permissions of files in the destination folder, such as removing
- execution permissions (:gh:`4662`, :gh:`4666`)
-
-
-.. _release-2.2.0:
-
-Scrapy 2.2.0 (2020-06-24)
--------------------------
-
-Highlights:
-
-* Python 3.5.2+ is required now
-* :ref:`dataclass objects <dataclass-items>` and
- :ref:`attrs objects <attrs-items>` are now valid :ref:`item types
- <item-types>`
-* New :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method
-* New :signal:`bytes_received` signal that allows canceling response download
-* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` fixes
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Support for Python 3.5.0 and 3.5.1 has been dropped; Scrapy now refuses to
- run with a Python version lower than 3.5.2, which introduced
- :class:`typing.Type` (:gh:`4615`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-* ``TextResponse.body_as_unicode()`` is now deprecated, use
- :attr:`TextResponse.text <scrapy.http.TextResponse.text>` instead
- (:gh:`4546`, :gh:`4555`, :gh:`4579`)
-
-* :class:`scrapy.item.BaseItem` is now deprecated, use
- :class:`scrapy.item.Item` instead (:gh:`4534`)
-
-
-New features
-~~~~~~~~~~~~
-
-* :ref:`dataclass objects <dataclass-items>` and
- :ref:`attrs objects <attrs-items>` are now valid :ref:`item types
- <item-types>`, and a new itemadapter_ library makes it easy to
- write code that :ref:`supports any item type <supporting-item-types>`
- (:gh:`2749`, :gh:`2807`, :gh:`3761`, :gh:`3881`, :gh:`4642`)
-
-* A new :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method
- allows to deserialize JSON responses (:gh:`2444`, :gh:`4460`,
- :gh:`4574`)
-
-* A new :signal:`bytes_received` signal allows monitoring response download
- progress and :ref:`stopping downloads <topics-stop-response-download>`
- (:gh:`4205`, :gh:`4559`)
-
-* The dictionaries in the result list of a :ref:`media pipeline
- <topics-media-pipeline>` now include a new key, ``status``, which indicates
- if the file was downloaded or, if the file was not downloaded, why it was
- not downloaded; see :meth:`FilesPipeline.get_media_requests
- <scrapy.pipelines.files.FilesPipeline.get_media_requests>` for more
- information (:gh:`2893`, :gh:`4486`)
-
-* When using :ref:`Google Cloud Storage <media-pipeline-gcs>` for
- a :ref:`media pipeline <topics-media-pipeline>`, a warning is now logged if
- the configured credentials do not grant the required permissions
- (:gh:`4346`, :gh:`4508`)
-
-* :ref:`Link extractors <topics-link-extractors>` are now serializable,
- as long as you do not use :ref:`lambdas <lambda>` for parameters; for
- example, you can now pass link extractors in :attr:`.Request.cb_kwargs`
- or :attr:`.Request.meta` when :ref:`persisting
- scheduled requests <topics-jobs>` (:gh:`4554`)
-
-* Upgraded the :ref:`pickle protocol <pickle-protocols>` that Scrapy uses
- from protocol 2 to protocol 4, improving serialization capabilities and
- performance (:gh:`4135`, :gh:`4541`)
-
-* :func:`scrapy.utils.misc.create_instance` now raises a :exc:`TypeError`
- exception if the resulting instance is ``None`` (:gh:`4528`,
- :gh:`4532`)
-
-.. _itemadapter: https://github.com/scrapy/itemadapter
-
-
-Bug fixes
-~~~~~~~~~
-
-* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
- discards cookies defined in :attr:`Request.headers
- <scrapy.Request.headers>` (:gh:`1992`, :gh:`2400`)
-
-* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
- re-encodes cookies defined as :class:`bytes` in the ``cookies`` parameter
- of the ``__init__`` method of :class:`~scrapy.Request`
- (:gh:`2400`, :gh:`3575`)
-
-* When :setting:`FEEDS` defines multiple URIs, :setting:`FEED_STORE_EMPTY` is
- ``False`` and the crawl yields no items, Scrapy no longer stops feed
- exports after the first URI (:gh:`4621`, :gh:`4626`)
-
-* :class:`~scrapy.spiders.Spider` callbacks defined using :doc:`coroutine
- syntax <topics/coroutines>` no longer need to return an iterable, and may
- instead return a :class:`~scrapy.Request` object, an
- :ref:`item <topics-items>`, or ``None`` (:gh:`4609`)
-
-* The :command:`startproject` command now ensures that the generated project
- folders and files have the right permissions (:gh:`4604`)
-
-* Fix a :exc:`KeyError` exception being sometimes raised from
- :class:`scrapy.utils.datatypes.LocalWeakReferencedCache` (:gh:`4597`,
- :gh:`4599`)
-
-* When :setting:`FEEDS` defines multiple URIs, log messages about items being
- stored now contain information from the corresponding feed, instead of
- always containing information about only one of the feeds (:gh:`4619`,
- :gh:`4629`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* Added a new section about :ref:`accessing cb_kwargs from errbacks
- <errback-cb_kwargs>` (:gh:`4598`, :gh:`4634`)
-
-* Covered chompjs_ in :ref:`topics-parsing-javascript` (:gh:`4556`,
- :gh:`4562`)
-
-* Removed from :doc:`topics/coroutines` the warning about the API being
- experimental (:gh:`4511`, :gh:`4513`)
-
-* Removed references to unsupported versions of :doc:`Twisted
- <twisted:index>` (:gh:`4533`)
-
-* Updated the description of the :ref:`screenshot pipeline example
- <ScreenshotPipeline>`, which now uses :doc:`coroutine syntax
- <topics/coroutines>` instead of returning a
- :class:`~twisted.internet.defer.Deferred` (:gh:`4514`, :gh:`4593`)
-
-* Removed a misleading import line from the
- :func:`scrapy.utils.log.configure_logging` code example (:gh:`4510`,
- :gh:`4587`)
-
-* The display-on-hover behavior of internal documentation references now also
- covers links to :ref:`commands <topics-commands>`, :attr:`.Request.meta`
- keys, :ref:`settings <topics-settings>` and
- :ref:`signals <topics-signals>` (:gh:`4495`, :gh:`4563`)
-
-* It is again possible to download the documentation for offline reading
- (:gh:`4578`, :gh:`4585`)
-
-* Removed backslashes preceding ``*args`` and ``**kwargs`` in some function
- and method signatures (:gh:`4592`, :gh:`4596`)
-
-.. _chompjs: https://github.com/Nykakin/chompjs
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-* Adjusted the code base further to our :ref:`style guidelines
- <coding-style>` (:gh:`4237`, :gh:`4525`, :gh:`4538`,
- :gh:`4539`, :gh:`4540`, :gh:`4542`, :gh:`4543`, :gh:`4544`,
- :gh:`4545`, :gh:`4557`, :gh:`4558`, :gh:`4566`, :gh:`4568`,
- :gh:`4572`)
-
-* Removed remnants of Python 2 support (:gh:`4550`, :gh:`4553`,
- :gh:`4568`)
-
-* Improved code sharing between the :command:`crawl` and :command:`runspider`
- commands (:gh:`4548`, :gh:`4552`)
-
-* Replaced ``chain(*iterable)`` with ``chain.from_iterable(iterable)``
- (:gh:`4635`)
-
-* You may now run the :mod:`asyncio` tests with Tox on any Python version
- (:gh:`4521`)
-
-* Updated test requirements to reflect an incompatibility with pytest 5.4 and
- 5.4.1 (:gh:`4588`)
-
-* Improved :class:`~scrapy.spiderloader.SpiderLoader` test coverage for
- scenarios involving duplicate spider names (:gh:`4549`, :gh:`4560`)
-
-* Configured Travis CI to also run the tests with Python 3.5.2
- (:gh:`4518`, :gh:`4615`)
-
-* Added a `Pylint <https://www.pylint.org/>`_ job to Travis CI
- (:gh:`3727`)
-
-* Added a `Mypy <https://mypy-lang.org/>`_ job to Travis CI (:gh:`4637`)
-
-* Made use of set literals in tests (:gh:`4573`)
-
-* Cleaned up the Travis CI configuration (:gh:`4517`, :gh:`4519`,
- :gh:`4522`, :gh:`4537`)
-
-
-.. _release-2.1.0:
-
-Scrapy 2.1.0 (2020-04-24)
--------------------------
-
-Highlights:
-
-* New :setting:`FEEDS` setting to export to multiple feeds
-* New :attr:`Response.ip_address <scrapy.http.Response.ip_address>` attribute
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* :exc:`AssertionError` exceptions triggered by :ref:`assert <assert>`
- statements have been replaced by new exception types, to support running
- Python in optimized mode (see :option:`-O`) without changing Scrapy’s
- behavior in any unexpected ways.
-
- If you catch an :exc:`AssertionError` exception from Scrapy, update your
- code to catch the corresponding new exception.
-
- (:gh:`4440`)
-
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-* The ``LOG_UNSERIALIZABLE_REQUESTS`` setting is no longer supported, use
- :setting:`SCHEDULER_DEBUG` instead (:gh:`4385`)
-
-* The ``REDIRECT_MAX_METAREFRESH_DELAY`` setting is no longer supported, use
- :setting:`METAREFRESH_MAXDELAY` instead (:gh:`4385`)
-
-* The :class:`~scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware`
- middleware has been removed, including the entire
- :class:`scrapy.downloadermiddlewares.chunked` module; chunked transfers
- work out of the box (:gh:`4431`)
-
-* The ``spiders`` property has been removed from
- :class:`~scrapy.crawler.Crawler`, use :class:`CrawlerRunner.spider_loader
- <scrapy.crawler.CrawlerRunner.spider_loader>` or instantiate
- :setting:`SPIDER_LOADER_CLASS` with your settings instead (:gh:`4398`)
-
-* The ``MultiValueDict``, ``MultiValueDictKeyError``, and ``SiteNode``
- classes have been removed from :mod:`scrapy.utils.datatypes`
- (:gh:`4400`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-* The ``FEED_FORMAT`` and ``FEED_URI`` settings have been deprecated in
- favor of the new :setting:`FEEDS` setting (:gh:`1336`, :gh:`3858`,
- :gh:`4507`)
-
-
-New features
-~~~~~~~~~~~~
-
-* A new setting, :setting:`FEEDS`, allows configuring multiple output feeds
- with different settings each (:gh:`1336`, :gh:`3858`, :gh:`4507`)
-
-* The :command:`crawl` and :command:`runspider` commands now support multiple
- ``-o`` parameters (:gh:`1336`, :gh:`3858`, :gh:`4507`)
-
-* The :command:`crawl` and :command:`runspider` commands now support
- specifying an output format by appending ``:<format>`` to the output file
- (:gh:`1336`, :gh:`3858`, :gh:`4507`)
-
-* The new :attr:`Response.ip_address <scrapy.http.Response.ip_address>`
- attribute gives access to the IP address that originated a response
- (:gh:`3903`, :gh:`3940`)
-
-* A warning is now issued when a value in
- :attr:`~scrapy.spiders.Spider.allowed_domains` includes a port
- (:gh:`50`, :gh:`3198`, :gh:`4413`)
-
-* Zsh completion now excludes used option aliases from the completion list
- (:gh:`4438`)
-
-
-Bug fixes
-~~~~~~~~~
-
-* :ref:`Request serialization <request-serialization>` no longer breaks for
- callbacks that are spider attributes which are assigned a function with a
- different name (:gh:`4500`)
-
-* ``None`` values in :attr:`~scrapy.spiders.Spider.allowed_domains` no longer
- cause a :exc:`TypeError` exception (:gh:`4410`)
-
-* Zsh completion no longer allows options after arguments (:gh:`4438`)
-
-* zope.interface 5.0.0 and later versions are now supported
- (:gh:`4447`, :gh:`4448`)
-
-* ``Spider.make_requests_from_url``, deprecated in Scrapy 1.4.0, now issues a
- warning when used (:gh:`4412`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* Improved the documentation about signals that allow their handlers to
- return a :class:`~twisted.internet.defer.Deferred` (:gh:`4295`,
- :gh:`4390`)
-
-* Our PyPI entry now includes links for our documentation, our source code
- repository and our issue tracker (:gh:`4456`)
-
-* Covered the `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_
- service in the documentation (:gh:`4206`, :gh:`4455`)
-
-* Removed references to the Guppy library, which only works in Python 2
- (:gh:`4285`, :gh:`4343`)
-
-* Extended use of InterSphinx to link to Python 3 documentation
- (:gh:`4444`, :gh:`4445`)
-
-* Added support for Sphinx 3.0 and later (:gh:`4475`, :gh:`4480`,
- :gh:`4496`, :gh:`4503`)
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-* Removed warnings about using old, removed settings (:gh:`4404`)
-
-* Removed a warning about importing
- :class:`~twisted.internet.testing.StringTransport` from
- ``twisted.test.proto_helpers`` in Twisted 19.7.0 or newer (:gh:`4409`)
-
-* Removed outdated Debian package build files (:gh:`4384`)
-
-* Removed :class:`object` usage as a base class (:gh:`4430`)
-
-* Removed code that added support for old versions of Twisted that we no
- longer support (:gh:`4472`)
-
-* Fixed code style issues (:gh:`4468`, :gh:`4469`, :gh:`4471`,
- :gh:`4481`)
-
-* Removed :func:`twisted.internet.defer.returnValue` calls (:gh:`4443`,
- :gh:`4446`, :gh:`4489`)
-
-
-.. _release-2.0.1:
-
-Scrapy 2.0.1 (2020-03-18)
--------------------------
-
-* :meth:`Response.follow_all <scrapy.http.Response.follow_all>` now supports
- an empty URL iterable as input (:gh:`4408`, :gh:`4420`)
-
-* Removed top-level :mod:`~twisted.internet.reactor` imports to prevent
- errors about the wrong Twisted reactor being installed when setting a
- different Twisted reactor using :setting:`TWISTED_REACTOR` (:gh:`4401`,
- :gh:`4406`)
-
-* Fixed tests (:gh:`4422`)
-
-
-.. _release-2.0.0:
-
-Scrapy 2.0.0 (2020-03-03)
--------------------------
-
-Highlights:
-
-* Python 2 support has been removed
-* :doc:`Partial <topics/coroutines>` :ref:`coroutine syntax <async>` support
- and :doc:`experimental <topics/asyncio>` :mod:`asyncio` support
-* New :meth:`Response.follow_all <scrapy.http.Response.follow_all>` method
-* :ref:`FTP support <media-pipeline-ftp>` for media pipelines
-* New :attr:`Response.certificate <scrapy.http.Response.certificate>`
- attribute
-* IPv6 support through ``DNS_RESOLVER``
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Python 2 support has been removed, following `Python 2 end-of-life on
- January 1, 2020`_ (:gh:`4091`, :gh:`4114`, :gh:`4115`,
- :gh:`4121`, :gh:`4138`, :gh:`4231`, :gh:`4242`, :gh:`4304`,
- :gh:`4309`, :gh:`4373`)
-
-* Retry gaveups (see :setting:`RETRY_TIMES`) are now logged as errors instead
- of as debug information (:gh:`3171`, :gh:`3566`)
-
-* File extensions that
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- ignores by default now also include ``7z``, ``7zip``, ``apk``, ``bz2``,
- ``cdr``, ``dmg``, ``ico``, ``iso``, ``tar``, ``tar.gz``, ``webm``, and
- ``xz`` (:gh:`1837`, :gh:`2067`, :gh:`4066`)
-
-* The :setting:`METAREFRESH_IGNORE_TAGS` setting is now an empty list by
- default, following web browser behavior (:gh:`3844`, :gh:`4311`)
-
-* The
- :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
- now includes spaces after commas in the value of the ``Accept-Encoding``
- header that it sets, following web browser behavior (:gh:`4293`)
-
-* The ``__init__`` method of custom download handlers (see
- :setting:`DOWNLOAD_HANDLERS`) or subclasses of the following downloader
- handlers no longer receives a ``settings`` parameter:
-
- * :class:`scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler`
-
- * :class:`scrapy.core.downloader.handlers.file.FileDownloadHandler`
-
- Use the ``from_settings`` or ``from_crawler`` class methods to expose such
- a parameter to your custom download handlers.
-
- (:gh:`4126`)
-
-* We have refactored the :class:`scrapy.core.scheduler.Scheduler` class and
- related queue classes (see :setting:`SCHEDULER_PRIORITY_QUEUE`,
- :setting:`SCHEDULER_DISK_QUEUE` and :setting:`SCHEDULER_MEMORY_QUEUE`) to
- make it easier to implement custom scheduler queue classes. See
- :ref:`2-0-0-scheduler-queue-changes` below for details.
-
-* Overridden settings are now logged in a different format. This is more in
- line with similar information logged at startup (:gh:`4199`)
-
-.. _Python 2 end-of-life on January 1, 2020: https://www.python.org/doc/sunset-python-2/
-
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-* The :ref:`Scrapy shell <topics-shell>` no longer provides a `sel` proxy
- object, use :meth:`response.selector <scrapy.http.TextResponse.selector>`
- instead (:gh:`4347`)
-
-* LevelDB support has been removed (:gh:`4112`)
-
-* The following functions have been removed from :mod:`scrapy.utils.python`:
- ``isbinarytext``, ``is_writable``, ``setattr_default``, ``stringify_dict``
- (:gh:`4362`)
-
-
-Deprecations
-~~~~~~~~~~~~
-
-* Using environment variables prefixed with ``SCRAPY_`` to override settings
- is deprecated (:gh:`4300`, :gh:`4374`, :gh:`4375`)
-
-* :class:`scrapy.linkextractors.FilteringLinkExtractor` is deprecated, use
- :class:`scrapy.linkextractors.LinkExtractor
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` instead (:gh:`4045`)
-
-* The ``noconnect`` query string argument of proxy URLs is deprecated and
- should be removed from proxy URLs (:gh:`4198`)
-
-* The :meth:`next <scrapy.utils.python.MutableChain.next>` method of
- :class:`scrapy.utils.python.MutableChain` is deprecated, use the global
- :func:`next` function or :meth:`MutableChain.__next__
- <scrapy.utils.python.MutableChain.__next__>` instead (:gh:`4153`)
-
-
-New features
-~~~~~~~~~~~~
-
-* Added :doc:`partial support <topics/coroutines>` for Python’s
- :ref:`coroutine syntax <async>` and :doc:`experimental support
- <topics/asyncio>` for :mod:`asyncio` and :mod:`asyncio`-powered libraries
- (:gh:`4010`, :gh:`4259`, :gh:`4269`, :gh:`4270`, :gh:`4271`,
- :gh:`4316`, :gh:`4318`)
-
-* The new :meth:`Response.follow_all <scrapy.http.Response.follow_all>`
- method offers the same functionality as
- :meth:`Response.follow <scrapy.http.Response.follow>` but supports an
- iterable of URLs as input and returns an iterable of requests
- (:gh:`2582`, :gh:`4057`, :gh:`4286`)
-
-* :ref:`Media pipelines <topics-media-pipeline>` now support :ref:`FTP
- storage <media-pipeline-ftp>` (:gh:`3928`, :gh:`3961`)
-
-* The new :attr:`Response.certificate <scrapy.http.Response.certificate>`
- attribute exposes the SSL certificate of the server as a
- :class:`twisted.internet.ssl.Certificate` object for HTTPS responses
- (:gh:`2726`, :gh:`4054`)
-
-* A new ``DNS_RESOLVER`` setting allows enabling IPv6 support
- (:gh:`1031`, :gh:`4227`)
-
-* A new :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` setting allows configuring
- the existing soft limit that pauses request downloads when the total
- response data being processed is too high (:gh:`1410`, :gh:`3551`)
-
-* A new :setting:`TWISTED_REACTOR` setting allows customizing the
- :mod:`~twisted.internet.reactor` that Scrapy uses, allowing to
- :doc:`enable asyncio support <topics/asyncio>` or deal with a
- :ref:`common macOS issue <faq-specific-reactor>` (:gh:`2905`,
- :gh:`4294`)
-
-* Scheduler disk and memory queues may now use the class methods
- ``from_crawler`` or ``from_settings`` (:gh:`3884`)
-
-* The new :attr:`Response.cb_kwargs <scrapy.http.Response.cb_kwargs>`
- attribute serves as a shortcut for :attr:`Response.request.cb_kwargs
- <scrapy.Request.cb_kwargs>` (:gh:`4331`)
-
-* :meth:`Response.follow <scrapy.http.Response.follow>` now supports a
- ``flags`` parameter, for consistency with :class:`~scrapy.Request`
- (:gh:`4277`, :gh:`4279`)
-
-* :ref:`Item loader processors <topics-loaders-processors>` can now be
- regular functions, they no longer need to be methods (:gh:`3899`)
-
-* :class:`~scrapy.spiders.Rule` now accepts an ``errback`` parameter
- (:gh:`4000`)
-
-* :class:`~scrapy.Request` no longer requires a ``callback`` parameter
- when an ``errback`` parameter is specified (:gh:`3586`, :gh:`4008`)
-
-* :class:`~scrapy.logformatter.LogFormatter` now supports some additional
- methods:
-
- * :class:`~scrapy.logformatter.LogFormatter.download_error` for
- download errors
-
- * :class:`~scrapy.logformatter.LogFormatter.item_error` for exceptions
- raised during item processing by :ref:`item pipelines
- <topics-item-pipeline>`
-
- * :class:`~scrapy.logformatter.LogFormatter.spider_error` for exceptions
- raised from :ref:`spider callbacks <topics-spiders>`
-
- (:gh:`374`, :gh:`3986`, :gh:`3989`, :gh:`4176`, :gh:`4188`)
-
-* The :setting:`FEED_URI` setting now supports :class:`pathlib.Path` values
- (:gh:`3731`, :gh:`4074`)
-
-* A new :signal:`request_left_downloader` signal is sent when a request
- leaves the downloader (:gh:`4303`)
-
-* Scrapy logs a warning when it detects a request callback or errback that
- uses ``yield`` but also returns a value, since the returned value would be
- lost (:gh:`3484`, :gh:`3869`)
-
-* :class:`~scrapy.spiders.Spider` objects now raise an :exc:`AttributeError`
- exception if they do not have a :class:`~scrapy.spiders.Spider.start_urls`
- attribute nor reimplement ``scrapy.spiders.Spider.start_requests()``,
- but have a ``start_url`` attribute (:gh:`4133`, :gh:`4170`)
-
-* :class:`~scrapy.exporters.BaseItemExporter` subclasses may now use
- ``super().__init__(**kwargs)`` instead of ``self._configure(kwargs)`` in
- their ``__init__`` method, passing ``dont_fail=True`` to the parent
- ``__init__`` method if needed, and accessing ``kwargs`` at ``self._kwargs``
- after calling their parent ``__init__`` method (:gh:`4193`,
- :gh:`4370`)
-
-* A new ``keep_fragments`` parameter of
- ``scrapy.utils.request.request_fingerprint`` allows to generate
- different fingerprints for requests with different fragments in their URL
- (:gh:`4104`)
-
-* Download handlers (see :setting:`DOWNLOAD_HANDLERS`) may now use the
- ``from_settings`` and ``from_crawler`` class methods that other Scrapy
- components already supported (:gh:`4126`)
-
-* :class:`scrapy.utils.python.MutableChain.__iter__` now returns ``self``,
- allowing it to be used as a sequence.
- (:gh:`4153`)
-
-
-Bug fixes
-~~~~~~~~~
-
-* The :command:`crawl` command now also exits with exit code 1 when an
- exception happens before the crawling starts (:gh:`4175`, :gh:`4207`)
-
-* :class:`LinkExtractor.extract_links
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` no longer
- re-encodes the query string or URLs from non-UTF-8 responses in UTF-8
- (:gh:`998`, :gh:`1403`, :gh:`1949`, :gh:`4321`)
-
-* The first spider middleware (see :setting:`SPIDER_MIDDLEWARES`) now also
- processes exceptions raised from callbacks that are generators
- (:gh:`4260`, :gh:`4272`)
-
-* Redirects to URLs starting with 3 slashes (``///``) are now supported
- (:gh:`4032`, :gh:`4042`)
-
-* :class:`~scrapy.Request` no longer accepts strings as ``url`` simply
- because they have a colon (:gh:`2552`, :gh:`4094`)
-
-* The correct encoding is now used for attach names in
- :class:`~scrapy.mail.MailSender` (:gh:`4229`, :gh:`4239`)
-
-* :class:`~scrapy.dupefilters.RFPDupeFilter`, the default
- :setting:`DUPEFILTER_CLASS`, no longer writes an extra ``\r`` character on
- each line in Windows, which made the size of the ``requests.seen`` file
- unnecessarily large on that platform (:gh:`4283`)
-
-* Z shell auto-completion now looks for ``.html`` files, not ``.http`` files,
- and covers the ``-h`` command-line switch (:gh:`4122`, :gh:`4291`)
-
-* Adding items to a :class:`scrapy.utils.datatypes.LocalCache` object
- without a ``limit`` defined no longer raises a :exc:`TypeError` exception
- (:gh:`4123`)
-
-* Fixed a typo in the message of the :exc:`ValueError` exception raised when
- :func:`scrapy.utils.misc.create_instance` gets both ``settings`` and
- ``crawler`` set to ``None`` (:gh:`4128`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* API documentation now links to an online, syntax-highlighted view of the
- corresponding source code (:gh:`4148`)
-
-* Links to unexisting documentation pages now allow access to the sidebar
- (:gh:`4152`, :gh:`4169`)
-
-* Cross-references within our documentation now display a tooltip when
- hovered (:gh:`4173`, :gh:`4183`)
-
-* Improved the documentation about :meth:`LinkExtractor.extract_links
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` and
- simplified :ref:`topics-link-extractors` (:gh:`4045`)
-
-* Clarified how :class:`ItemLoader.item <scrapy.loader.ItemLoader.item>`
- works (:gh:`3574`, :gh:`4099`)
-
-* Clarified that :func:`logging.basicConfig` should not be used when also
- using :class:`~scrapy.crawler.CrawlerProcess` (:gh:`2149`,
- :gh:`2352`, :gh:`3146`, :gh:`3960`)
-
-* Clarified the requirements for :class:`~scrapy.Request` objects
- :ref:`when using persistence <request-serialization>` (:gh:`4124`,
- :gh:`4139`)
-
-* Clarified how to install a :ref:`custom image pipeline
- <media-pipeline-example>` (:gh:`4034`, :gh:`4252`)
-
-* Fixed the signatures of the ``file_path`` method in :ref:`media pipeline
- <topics-media-pipeline>` examples (:gh:`4290`)
-
-* Covered a backward-incompatible change in Scrapy 1.7.0 affecting custom
- :class:`scrapy.core.scheduler.Scheduler` subclasses (:gh:`4274`)
-
-* Improved the ``README.rst`` and ``CODE_OF_CONDUCT.md`` files
- (:gh:`4059`)
-
-* Documentation examples are now checked as part of our test suite and we
- have fixed some of the issues detected (:gh:`4142`, :gh:`4146`,
- :gh:`4171`, :gh:`4184`, :gh:`4190`)
-
-* Fixed logic issues, broken links and typos (:gh:`4247`, :gh:`4258`,
- :gh:`4282`, :gh:`4288`, :gh:`4305`, :gh:`4308`, :gh:`4323`,
- :gh:`4338`, :gh:`4359`, :gh:`4361`)
-
-* Improved consistency when referring to the ``__init__`` method of an object
- (:gh:`4086`, :gh:`4088`)
-
-* Fixed an inconsistency between code and output in :ref:`intro-overview`
- (:gh:`4213`)
-
-* Extended :mod:`~sphinx.ext.intersphinx` usage (:gh:`4147`,
- :gh:`4172`, :gh:`4185`, :gh:`4194`, :gh:`4197`)
-
-* We now use a recent version of Python to build the documentation
- (:gh:`4140`, :gh:`4249`)
-
-* Cleaned up documentation (:gh:`4143`, :gh:`4275`)
-
-
-Quality assurance
-~~~~~~~~~~~~~~~~~
-
-* Re-enabled proxy ``CONNECT`` tests (:gh:`2545`, :gh:`4114`)
-
-* Added Bandit_ security checks to our test suite (:gh:`4162`,
- :gh:`4181`)
-
-* Added Flake8_ style checks to our test suite and applied many of the
- corresponding changes (:gh:`3944`, :gh:`3945`, :gh:`4137`,
- :gh:`4157`, :gh:`4167`, :gh:`4174`, :gh:`4186`, :gh:`4195`,
- :gh:`4238`, :gh:`4246`, :gh:`4355`, :gh:`4360`, :gh:`4365`)
-
-* Improved test coverage (:gh:`4097`, :gh:`4218`, :gh:`4236`)
-
-* Started reporting slowest tests, and improved the performance of some of
- them (:gh:`4163`, :gh:`4164`)
-
-* Fixed broken tests and refactored some tests (:gh:`4014`, :gh:`4095`,
- :gh:`4244`, :gh:`4268`, :gh:`4372`)
-
-* Modified the :doc:`tox <tox:index>` configuration to allow running tests
- with any Python version, run Bandit_ and Flake8_ tests by default, and
- enforce a minimum tox version programmatically (:gh:`4179`)
-
-* Cleaned up code (:gh:`3937`, :gh:`4208`, :gh:`4209`,
- :gh:`4210`, :gh:`4212`, :gh:`4369`, :gh:`4376`, :gh:`4378`)
-
-.. _Bandit: https://bandit.readthedocs.io/en/latest/
-.. _Flake8: https://flake8.pycqa.org/en/latest/
-
-
-.. _2-0-0-scheduler-queue-changes:
-
-Changes to scheduler queue classes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following changes may impact any custom queue classes of all types:
-
-* The ``push`` method no longer receives a second positional parameter
- containing ``request.priority * -1``. If you need that value, get it
- from the first positional parameter, ``request``, instead, or use
- the new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
- method in :class:`scrapy.core.scheduler.ScrapyPriorityQueue`
- subclasses.
-
-The following changes may impact custom priority queue classes:
-
-* In the ``__init__`` method or the ``from_crawler`` or ``from_settings``
- class methods:
-
- * The parameter that used to contain a factory function,
- ``qfactory``, is now passed as a keyword parameter named
- ``downstream_queue_cls``.
-
- * A new keyword parameter has been added: ``key``. It is a string
- that is always an empty string for memory queues and indicates the
- :setting:`JOBDIR` value for disk queues.
-
- * The parameter for disk queues that contains data from the previous
- crawl, ``startprios`` or ``slot_startprios``, is now passed as a
- keyword parameter named ``startprios``.
-
- * The ``serialize`` parameter is no longer passed. The disk queue
- class must take care of request serialization on its own before
- writing to disk, using the
- :func:`~scrapy.utils.reqser.request_to_dict` and
- :func:`~scrapy.utils.reqser.request_from_dict` functions from the
- :mod:`scrapy.utils.reqser` module.
-
-The following changes may impact custom disk and memory queue classes:
-
-* The signature of the ``__init__`` method is now
- ``__init__(self, crawler, key)``.
-
-The following changes affect specifically the
-:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` and
-:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` classes from
-:mod:`scrapy.core.scheduler` and may affect subclasses:
-
-* In the ``__init__`` method, most of the changes described above apply.
-
- ``__init__`` may still receive all parameters as positional parameters,
- however:
-
- * ``downstream_queue_cls``, which replaced ``qfactory``, must be
- instantiated differently.
-
- ``qfactory`` was instantiated with a priority value (integer).
-
- Instances of ``downstream_queue_cls`` should be created using
- the new
- :meth:`ScrapyPriorityQueue.qfactory <scrapy.core.scheduler.ScrapyPriorityQueue.qfactory>`
- or
- :meth:`DownloaderAwarePriorityQueue.pqfactory <scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqfactory>`
- methods.
-
- * The new ``key`` parameter displaced the ``startprios``
- parameter 1 position to the right.
-
-* The following class attributes have been added:
-
- * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.crawler`
-
- * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.downstream_queue_cls`
- (details above)
-
- * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.key` (details above)
-
-* The ``serialize`` attribute has been removed (details above)
-
-The following changes affect specifically the
-:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` class and may affect
-subclasses:
-
-* A new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
- method has been added which, given a request, returns
- ``request.priority * -1``.
-
- It is used in :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.push`
- to make up for the removal of its ``priority`` parameter.
-
-* The ``spider`` attribute has been removed. Use
- :attr:`crawler.spider <scrapy.core.scheduler.ScrapyPriorityQueue.crawler>`
- instead.
-
-The following changes affect specifically the
-:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` class and may
-affect subclasses:
-
-* A new :attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqueues`
- attribute offers a mapping of downloader slot names to the
- corresponding instances of
- :attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.downstream_queue_cls`.
-
-(:gh:`3884`)
-
-.. _release-1.8.4:
-
-Scrapy 1.8.4 (2024-02-14)
--------------------------
-
-**Security bug fixes:**
-
-- Due to its `ReDoS vulnerabilities`_, ``scrapy.utils.iterators.xmliter`` is
- now deprecated in favor of :func:`~scrapy.utils.iterators.xmliter_lxml`,
- which :class:`~scrapy.spiders.XMLFeedSpider` now uses.
-
- To minimize the impact of this change on existing code,
- :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
- the node namespace as a prefix in the node name, and big files with highly
- nested trees when using libxml2 2.7+.
-
- Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
-
-- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
- to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
- advisory`_ for more information.
-
-- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, use of the
- ``scrapy.downloadermiddlewares.decompression`` module is discouraged and
- will trigger a warning.
-
-- The ``Authorization`` header is now dropped on redirects to a different
- domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
- information.
-
- .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
-
-
-.. _release-1.8.3:
-
-Scrapy 1.8.3 (2022-07-25)
--------------------------
-
-**Security bug fix:**
-
-- When :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
- processes a request with :reqmeta:`proxy` metadata, and that
- :reqmeta:`proxy` metadata includes proxy credentials,
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` sets
- the ``Proxy-Authorization`` header, but only if that header is not already
- set.
-
- There are third-party proxy-rotation downloader middlewares that set
- different :reqmeta:`proxy` metadata every time they process a request.
-
- Because of request retries and redirects, the same request can be processed
- by downloader middlewares more than once, including both
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and
- any third-party proxy-rotation downloader middleware.
-
- These third-party proxy-rotation downloader middlewares could change the
- :reqmeta:`proxy` metadata of a request to a new value, but fail to remove
- the ``Proxy-Authorization`` header from the previous value of the
- :reqmeta:`proxy` metadata, causing the credentials of one proxy to be sent
- to a different proxy.
-
- To prevent the unintended leaking of proxy credentials, the behavior of
- :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` is now
- as follows when processing a request:
-
- - If the request being processed defines :reqmeta:`proxy` metadata that
- includes credentials, the ``Proxy-Authorization`` header is always
- updated to feature those credentials.
-
- - If the request being processed defines :reqmeta:`proxy` metadata
- without credentials, the ``Proxy-Authorization`` header is removed
- *unless* it was originally defined for the same proxy URL.
-
- To remove proxy credentials while keeping the same proxy URL, remove
- the ``Proxy-Authorization`` header.
-
- - If the request has no :reqmeta:`proxy` metadata, or that metadata is a
- falsy value (e.g. ``None``), the ``Proxy-Authorization`` header is
- removed.
-
- It is no longer possible to set a proxy URL through the
- :reqmeta:`proxy` metadata but set the credentials through the
- ``Proxy-Authorization`` header. Set proxy credentials through the
- :reqmeta:`proxy` metadata instead.
-
-
-.. _release-1.8.2:
-
-Scrapy 1.8.2 (2022-03-01)
--------------------------
-
-**Security bug fixes:**
-
-- When a :class:`~scrapy.Request` object with cookies defined gets a
- redirect response causing a new :class:`~scrapy.Request` object to be
- scheduled, the cookies defined in the original
- :class:`~scrapy.Request` object are no longer copied into the new
- :class:`~scrapy.Request` object.
-
- If you manually set the ``Cookie`` header on a
- :class:`~scrapy.Request` object and the domain name of the redirect
- URL is not an exact match for the domain of the URL of the original
- :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
- from the new :class:`~scrapy.Request` object.
-
- The old behavior could be exploited by an attacker to gain access to your
- cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
- information.
-
- .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8
-
- .. note:: It is still possible to enable the sharing of cookies between
- different domains with a shared domain suffix (e.g.
- ``example.com`` and any subdomain) by defining the shared domain
- suffix (e.g. ``example.com``) as the cookie domain when defining
- your cookies. See the documentation of the
- :class:`~scrapy.Request` class for more information.
-
-- When the domain of a cookie, either received in the ``Set-Cookie`` header
- of a response or defined in a :class:`~scrapy.Request` object, is set
- to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
- ignored unless the cookie domain is the same as the request domain.
-
- The old behavior could be exploited by an attacker to inject cookies into
- your requests to some other domains. Please, see the `mfjm-vh54-3f96
- security advisory`_ for more information.
-
- .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96
-
-
-.. _release-1.8.1:
-
-Scrapy 1.8.1 (2021-10-05)
--------------------------
-
-* **Security bug fix:**
-
- If you use
- :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
- (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP
- authentication, any request exposes your credentials to the request target.
-
- To prevent unintended exposure of authentication credentials to unintended
- domains, you must now additionally set a new, additional spider attribute,
- ``http_auth_domain``, and point it to the specific domain to which the
- authentication credentials must be sent.
-
- If the ``http_auth_domain`` spider attribute is not set, the domain of the
- first request will be considered the HTTP authentication target, and
- authentication credentials will only be sent in requests targeting that
- domain.
-
- If you need to send the same HTTP authentication credentials to multiple
- domains, you can use :func:`w3lib.http.basic_auth_header` instead to
- set the value of the ``Authorization`` header of your requests.
-
- If you *really* want your spider to send the same HTTP authentication
- credentials to any domain, set the ``http_auth_domain`` spider attribute
- to ``None``.
-
- Finally, if you are a user of `scrapy-splash`_, know that this version of
- Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will
- need to upgrade scrapy-splash to a greater version for it to continue to
- work.
-
-.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
-
-
-.. _release-1.8.0:
-
-Scrapy 1.8.0 (2019-10-28)
--------------------------
-
-Highlights:
-
-* Dropped Python 3.4 support and updated minimum requirements; made Python 3.8
- support official
-* New :meth:`.Request.from_curl` class method
-* New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings
-* New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
- :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. skip: start
-
-* Python 3.4 is no longer supported, and some of the minimum requirements of
- Scrapy have also changed:
-
- * :doc:`cssselect <cssselect:index>` 0.9.1
- * cryptography_ 2.0
- * lxml_ 3.5.0
- * pyOpenSSL_ 16.2.0
- * queuelib_ 1.4.2
- * service_identity_ 16.0.0
- * six_ 1.10.0
- * Twisted_ 17.9.0 (16.0.0 with Python 2)
- * zope.interface_ 4.1.3
-
- (:gh:`3892`)
-
-* ``JSONRequest`` is now called :class:`~scrapy.http.JsonRequest` for
- consistency with similar classes (:gh:`3929`, :gh:`3982`)
-
-* If you are using a custom context factory
- (``DOWNLOADER_CLIENTCONTEXTFACTORY``), its ``__init__`` method must
- accept two new parameters: ``tls_verbose_logging`` and ``tls_ciphers``
- (:gh:`2111`, :gh:`3392`, :gh:`3442`, :gh:`3450`)
-
-* :class:`~scrapy.loader.ItemLoader` now turns the values of its input item
- into lists:
-
- .. code-block:: pycon
-
- >>> item = MyItem()
- >>> item["field"] = "value1"
- >>> loader = ItemLoader(item=item)
- >>> item["field"]
- ['value1']
-
- This is needed to allow adding values to existing fields
- (``loader.add_value('field', 'value2')``).
-
- (:gh:`3804`, :gh:`3819`, :gh:`3897`, :gh:`3976`, :gh:`3998`,
- :gh:`4036`)
-
-.. skip: end
-
-See also :ref:`1.8-deprecation-removals` below.
-
-
-New features
-~~~~~~~~~~~~
-
-* A new :meth:`Request.from_curl <scrapy.Request.from_curl>` class
- method allows :ref:`creating a request from a cURL command
- <requests-from-curl>` (:gh:`2985`, :gh:`3862`)
-
-* A new :setting:`ROBOTSTXT_PARSER` setting allows choosing which robots.txt_
- parser to use. It includes built-in support for
- :ref:`RobotFileParser <python-robotfileparser>`,
- :ref:`Protego <protego-parser>` (default), Reppy, and
- :ref:`Robotexclusionrulesparser <rerp-parser>`, and allows you to
- :ref:`implement support for additional parsers
- <support-for-new-robots-parser>` (:gh:`754`, :gh:`2669`,
- :gh:`3796`, :gh:`3935`, :gh:`3969`, :gh:`4006`)
-
-* A new :setting:`ROBOTSTXT_USER_AGENT` setting allows defining a separate
- user agent string to use for robots.txt_ parsing (:gh:`3931`,
- :gh:`3966`)
-
-* :class:`~scrapy.spiders.Rule` no longer requires a :class:`LinkExtractor
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` parameter
- (:gh:`781`, :gh:`4016`)
-
-* Use the new :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting to customize
- the TLS/SSL ciphers used by the default HTTP/1.1 downloader (:gh:`3392`,
- :gh:`3442`)
-
-* Set the new :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` setting to
- ``True`` to enable debug-level messages about TLS connection parameters
- after establishing HTTPS connections (:gh:`2111`, :gh:`3450`)
-
-* Callbacks that receive keyword arguments (see :attr:`.Request.cb_kwargs`)
- can now be tested using the new :class:`@cb_kwargs
- <scrapy.contracts.default.CallbackKeywordArgumentsContract>`
- :ref:`spider contract <topics-contracts>` (:gh:`3985`, :gh:`3988`)
-
-* When a :class:`@scrapes <scrapy.contracts.default.ScrapesContract>` spider
- contract fails, all missing fields are now reported (:gh:`766`,
- :gh:`3939`)
-
-* :ref:`Custom log formats <custom-log-formats>` can now drop messages by
- having the corresponding methods of the configured :setting:`LOG_FORMATTER`
- return ``None`` (:gh:`3984`, :gh:`3987`)
-
-* A much improved completion definition is now available for Zsh_
- (:gh:`4069`)
-
-
-Bug fixes
-~~~~~~~~~
-
-* :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` no
- longer makes later calls to :meth:`ItemLoader.get_output_value()
- <scrapy.loader.ItemLoader.get_output_value>` or
- :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` return
- empty data (:gh:`3804`, :gh:`3819`, :gh:`3897`, :gh:`3976`,
- :gh:`3998`, :gh:`4036`)
-
-* Fixed :class:`~scrapy.statscollectors.DummyStatsCollector` raising a
- :exc:`TypeError` exception (:gh:`4007`, :gh:`4052`)
-
-* :meth:`FilesPipeline.file_path
- <scrapy.pipelines.files.FilesPipeline.file_path>` and
- :meth:`ImagesPipeline.file_path
- <scrapy.pipelines.images.ImagesPipeline.file_path>` no longer choose
- file extensions that are not `registered with IANA`_ (:gh:`1287`,
- :gh:`3953`, :gh:`3954`)
-
-* When using botocore_ to persist files in S3, all botocore-supported headers
- are properly mapped now (:gh:`3904`, :gh:`3905`)
-
-* FTP passwords in :setting:`FEED_URI` containing percent-escaped characters
- are now properly decoded (:gh:`3941`)
-
-* A memory-handling and error-handling issue in
- ``scrapy.utils.ssl.get_temp_key_info()`` has been fixed (:gh:`3920`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* The documentation now covers how to define and configure a :ref:`custom log
- format <custom-log-formats>` (:gh:`3616`, :gh:`3660`)
-
-* API documentation added for :class:`~scrapy.exporters.MarshalItemExporter`
- and :class:`~scrapy.exporters.PythonItemExporter` (:gh:`3973`)
-
-* API documentation added for :class:`~scrapy.item.BaseItem` and
- :class:`~scrapy.item.ItemMeta` (:gh:`3999`)
-
-* Minor documentation fixes (:gh:`2998`, :gh:`3398`, :gh:`3597`,
- :gh:`3894`, :gh:`3934`, :gh:`3978`, :gh:`3993`, :gh:`4022`,
- :gh:`4028`, :gh:`4033`, :gh:`4046`, :gh:`4050`, :gh:`4055`,
- :gh:`4056`, :gh:`4061`, :gh:`4072`, :gh:`4071`, :gh:`4079`,
- :gh:`4081`, :gh:`4089`, :gh:`4093`)
-
-
-.. _1.8-deprecation-removals:
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-* ``scrapy.xlib`` has been removed (:gh:`4015`)
-
-
-.. _1.8-deprecations:
-
-Deprecations
-~~~~~~~~~~~~
-
-* The LevelDB_ storage backend
- (``scrapy.extensions.httpcache.LeveldbCacheStorage``) of
- :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` is
- deprecated (:gh:`4085`, :gh:`4092`)
-
-* Use of the undocumented ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
- variable is deprecated (:gh:`3910`)
-
-* ``scrapy.item.DictItem`` is deprecated, use :class:`~scrapy.item.Item`
- instead (:gh:`3999`)
-
-
-Other changes
-~~~~~~~~~~~~~
-
-* Minimum versions of optional Scrapy requirements that are covered by
- continuous integration tests have been updated:
-
- * botocore_ 1.3.23
- * Pillow_ 3.4.2
-
- Lower versions of these optional requirements may work, but it is not
- guaranteed (:gh:`3892`)
-
-* GitHub templates for bug reports and feature requests (:gh:`3126`,
- :gh:`3471`, :gh:`3749`, :gh:`3754`)
-
-* Continuous integration fixes (:gh:`3923`)
-
-* Code cleanup (:gh:`3391`, :gh:`3907`, :gh:`3946`, :gh:`3950`,
- :gh:`4023`, :gh:`4031`)
-
-
-.. _release-1.7.4:
-
-Scrapy 1.7.4 (2019-10-21)
--------------------------
-
-Revert the fix for :gh:`3804` (:gh:`3819`), which has a few undesired
-side effects (:gh:`3897`, :gh:`3976`).
-
-As a result, when an item loader is initialized with an item,
-:meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` once again
-makes later calls to :meth:`ItemLoader.get_output_value()
-<scrapy.loader.ItemLoader.get_output_value>` or :meth:`ItemLoader.load_item()
-<scrapy.loader.ItemLoader.load_item>` return empty data.
-
-
-.. _release-1.7.3:
-
-Scrapy 1.7.3 (2019-08-01)
--------------------------
-
-Enforce lxml 4.3.5 or lower for Python 3.4 (:gh:`3912`, :gh:`3918`).
-
-
-.. _release-1.7.2:
-
-Scrapy 1.7.2 (2019-07-23)
--------------------------
-
-Fix Python 2 support (:gh:`3889`, :gh:`3893`, :gh:`3896`).
-
-
-.. _release-1.7.1:
-
-Scrapy 1.7.1 (2019-07-18)
--------------------------
-
-Re-packaging of Scrapy 1.7.0, which was missing some changes in PyPI.
-
-
-.. _release-1.7.0:
-
-Scrapy 1.7.0 (2019-07-18)
--------------------------
-
-.. note:: Make sure you install Scrapy 1.7.1. The Scrapy 1.7.0 package in PyPI
- is the result of an erroneous commit tagging and does not include all
- the changes described below.
-
-Highlights:
-
-* Improvements for crawls targeting multiple domains
-* A cleaner way to pass arguments to callbacks
-* A new class for JSON requests
-* Improvements for rule-based spiders
-* New features for feed exports
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* ``429`` is now part of the :setting:`RETRY_HTTP_CODES` setting by default
-
- This change is **backward incompatible**. If you don’t want to retry
- ``429``, you must override :setting:`RETRY_HTTP_CODES` accordingly.
-
-* :class:`~scrapy.crawler.Crawler`,
- :meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
- :meth:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
- no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, they
- only accept a :class:`~scrapy.spiders.Spider` subclass now.
-
- :class:`~scrapy.spiders.Spider` subclass instances were never meant to
- work, and they were not working as one would expect: instead of using the
- passed :class:`~scrapy.spiders.Spider` subclass instance, their
- :class:`~scrapy.spiders.Spider.from_crawler` method was called to generate
- a new instance.
-
-* Non-default values for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting
- may stop working. Scheduler priority queue classes now need to handle
- :class:`~scrapy.Request` objects instead of arbitrary Python data
- structures.
-
-* An additional ``crawler`` parameter has been added to the ``__init__``
- method of the :class:`~scrapy.core.scheduler.Scheduler` class. Custom
- scheduler subclasses which don't accept arbitrary parameters in their
- ``__init__`` method might break because of this change.
-
- For more information, see :setting:`SCHEDULER`.
-
-See also :ref:`1.7-deprecation-removals` below.
-
-
-New features
-~~~~~~~~~~~~
-
-* A new scheduler priority queue,
- ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
- :ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
- scheduling improvement on crawls targeting multiple web domains, at the
- cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:gh:`3520`)
-
-* A new :attr:`.Request.cb_kwargs` attribute
- provides a cleaner way to pass keyword arguments to callback methods
- (:gh:`1138`, :gh:`3563`)
-
-* A new :class:`JSONRequest <scrapy.http.JsonRequest>` class offers a more
- convenient way to build JSON requests (:gh:`3504`, :gh:`3505`)
-
-* A ``process_request`` callback passed to the :class:`~scrapy.spiders.Rule`
- ``__init__`` method now receives the :class:`~scrapy.http.Response` object that
- originated the request as its second argument (:gh:`3682`)
-
-* A new ``restrict_text`` parameter for the
- :attr:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- ``__init__`` method allows filtering links by linking text (:gh:`3622`,
- :gh:`3635`)
-
-* A new :setting:`FEED_STORAGE_S3_ACL` setting allows defining a custom ACL
- for feeds exported to Amazon S3 (:gh:`3607`)
-
-* A new :setting:`FEED_STORAGE_FTP_ACTIVE` setting allows using FTP’s active
- connection mode for feeds exported to FTP servers (:gh:`3829`)
-
-* A new :setting:`METAREFRESH_IGNORE_TAGS` setting allows overriding which
- HTML tags are ignored when searching a response for HTML meta tags that
- trigger a redirect (:gh:`1422`, :gh:`3768`)
-
-* A new :reqmeta:`redirect_reasons` request meta key exposes the reason
- (status code, meta refresh) behind every followed redirect (:gh:`3581`,
- :gh:`3687`)
-
-* The ``SCRAPY_CHECK`` variable is now set to the ``true`` string during runs
- of the :command:`check` command, which allows :ref:`detecting contract
- check runs from code <detecting-contract-check-runs>` (:gh:`3704`,
- :gh:`3739`)
-
-* A new :meth:`Item.deepcopy() <scrapy.item.Item.deepcopy>` method makes it
- easier to :ref:`deep-copy items <copying-items>` (:gh:`1493`,
- :gh:`3671`)
-
-* :class:`~scrapy.extensions.corestats.CoreStats` also logs
- ``elapsed_time_seconds`` now (:gh:`3638`)
-
-* Exceptions from :class:`~scrapy.loader.ItemLoader` :ref:`input and output
- processors <topics-loaders-processors>` are now more verbose
- (:gh:`3836`, :gh:`3840`)
-
-* :class:`~scrapy.crawler.Crawler`,
- :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
- :class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
- now fail gracefully if they receive a :class:`~scrapy.spiders.Spider`
- subclass instance instead of the subclass itself (:gh:`2283`,
- :gh:`3610`, :gh:`3872`)
-
-
-Bug fixes
-~~~~~~~~~
-
-* :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
- is now also invoked for generators (:gh:`220`, :gh:`2061`)
-
-* System exceptions like KeyboardInterrupt_ are no longer caught
- (:gh:`3726`)
-
-* :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` no
- longer makes later calls to :meth:`ItemLoader.get_output_value()
- <scrapy.loader.ItemLoader.get_output_value>` or
- :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` return
- empty data (:gh:`3804`, :gh:`3819`)
-
-* The images pipeline (:class:`~scrapy.pipelines.images.ImagesPipeline`) no
- longer ignores these Amazon S3 settings: :setting:`AWS_ENDPOINT_URL`,
- :setting:`AWS_REGION_NAME`, :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`
- (:gh:`3625`)
-
-* Fixed a memory leak in ``scrapy.pipelines.media.MediaPipeline`` affecting,
- for example, non-200 responses and exceptions from custom middlewares
- (:gh:`3813`)
-
-* Requests with private callbacks are now correctly unserialized from disk
- (:gh:`3790`)
-
-* :meth:`.FormRequest.from_response`
- now handles invalid methods like major web browsers (:gh:`3777`,
- :gh:`3794`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-* A new topic, :ref:`topics-dynamic-content`, covers recommended approaches
- to read dynamically-loaded data (:gh:`3703`)
-
-* :ref:`topics-broad-crawls` now features information about memory usage
- (:gh:`1264`, :gh:`3866`)
-
-* The documentation of :class:`~scrapy.spiders.Rule` now covers how to access
- the text of a link when using :class:`~scrapy.spiders.CrawlSpider`
- (:gh:`3711`, :gh:`3712`)
-
-* A new section, :ref:`httpcache-storage-custom`, covers writing a custom
- cache storage backend for
- :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
- (:gh:`3683`, :gh:`3692`)
-
-* A new :ref:`FAQ <faq>` entry, :ref:`faq-split-item`, explains what to do
- when you want to split an item into multiple items from an item pipeline
- (:gh:`2240`, :gh:`3672`)
-
-* Updated the :ref:`FAQ entry about crawl order <faq-bfo-dfo>` to explain why
- the first few requests rarely follow the desired order (:gh:`1739`,
- :gh:`3621`)
-
-* The :setting:`LOGSTATS_INTERVAL` setting (:gh:`3730`), the
- :meth:`FilesPipeline.file_path <scrapy.pipelines.files.FilesPipeline.file_path>`
- and
- :meth:`ImagesPipeline.file_path <scrapy.pipelines.images.ImagesPipeline.file_path>`
- methods (:gh:`2253`, :gh:`3609`) and the
- :meth:`Crawler.stop() <scrapy.crawler.Crawler.stop>` method (:gh:`3842`)
- are now documented
-
-* Some parts of the documentation that were confusing or misleading are now
- clearer (:gh:`1347`, :gh:`1789`, :gh:`2289`, :gh:`3069`,
- :gh:`3615`, :gh:`3626`, :gh:`3668`, :gh:`3670`, :gh:`3673`,
- :gh:`3728`, :gh:`3762`, :gh:`3861`, :gh:`3882`)
-
-* Minor documentation fixes (:gh:`3648`, :gh:`3649`, :gh:`3662`,
- :gh:`3674`, :gh:`3676`, :gh:`3694`, :gh:`3724`, :gh:`3764`,
- :gh:`3767`, :gh:`3791`, :gh:`3797`, :gh:`3806`, :gh:`3812`)
-
-.. _1.7-deprecation-removals:
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-The following deprecated APIs have been removed (:gh:`3578`):
-
-* ``scrapy.conf`` (use :attr:`Crawler.settings
- <scrapy.crawler.Crawler.settings>`)
-
-* From ``scrapy.core.downloader.handlers``:
-
- * ``http.HttpDownloadHandler`` (use ``http10.HTTP10DownloadHandler``)
-
-* ``scrapy.loader.ItemLoader._get_values`` (use ``_get_xpathvalues``)
-
-* ``scrapy.loader.XPathItemLoader`` (use :class:`~scrapy.loader.ItemLoader`)
-
-* ``scrapy.log`` (see :ref:`topics-logging`)
-
-* From ``scrapy.pipelines``:
-
- * ``files.FilesPipeline.file_key`` (use ``file_path``)
-
- * ``images.ImagesPipeline.file_key`` (use ``file_path``)
-
- * ``images.ImagesPipeline.image_key`` (use ``file_path``)
-
- * ``images.ImagesPipeline.thumb_key`` (use ``thumb_path``)
-
-* From both ``scrapy.selector`` and ``scrapy.selector.lxmlsel``:
-
- * ``HtmlXPathSelector`` (use :class:`~scrapy.Selector`)
-
- * ``XmlXPathSelector`` (use :class:`~scrapy.Selector`)
-
- * ``XPathSelector`` (use :class:`~scrapy.Selector`)
-
- * ``XPathSelectorList`` (use :class:`~scrapy.Selector`)
-
-* From ``scrapy.selector.csstranslator``:
-
- * ``ScrapyGenericTranslator`` (use parsel.csstranslator.GenericTranslator_)
-
- * ``ScrapyHTMLTranslator`` (use parsel.csstranslator.HTMLTranslator_)
-
- * ``ScrapyXPathExpr`` (use parsel.csstranslator.XPathExpr_)
-
-* From :class:`~scrapy.Selector`:
-
- * ``_root`` (both the ``__init__`` method argument and the object property, use
- ``root``)
-
- * ``extract_unquoted`` (use ``getall``)
-
- * ``select`` (use ``xpath``)
-
-* From :class:`~scrapy.selector.SelectorList`:
-
- * ``extract_unquoted`` (use ``getall``)
-
- * ``select`` (use ``xpath``)
-
- * ``x`` (use ``xpath``)
-
-* ``scrapy.spiders.BaseSpider`` (use :class:`~scrapy.spiders.Spider`)
-
-* From :class:`~scrapy.spiders.Spider` (and subclasses):
-
- * ``DOWNLOAD_DELAY`` (use :ref:`download_delay
- <spider-download_delay-attribute>`)
-
- * ``set_crawler`` (use :meth:`~scrapy.spiders.Spider.from_crawler`)
-
-* ``scrapy.spiders.spiders`` (use :class:`~scrapy.spiderloader.SpiderLoader`)
-
-* ``scrapy.telnet`` (use :mod:`scrapy.extensions.telnet`)
-
-* From ``scrapy.utils.python``:
-
- * ``str_to_unicode`` (use ``to_unicode``)
-
- * ``unicode_to_str`` (use ``to_bytes``)
-
-* ``scrapy.utils.response.body_or_str``
-
-The following deprecated settings have also been removed (:gh:`3578`):
-
-* ``SPIDER_MANAGER_CLASS`` (use :setting:`SPIDER_LOADER_CLASS`)
-
-
-.. _1.7-deprecations:
-
-Deprecations
-~~~~~~~~~~~~
-
-* The ``queuelib.PriorityQueue`` value for the
- :setting:`SCHEDULER_PRIORITY_QUEUE` setting is deprecated. Use
- ``scrapy.pqueues.ScrapyPriorityQueue`` instead.
-
-* ``process_request`` callbacks passed to :class:`~scrapy.spiders.Rule` that
- do not accept two arguments are deprecated.
-
-* The following modules are deprecated:
-
- * ``scrapy.utils.http`` (use `w3lib.http`_)
+.. Once the first release of a minor version series is over a year old,
+ move the whole series to news/2.x.rst.
- * ``scrapy.utils.markup`` (use `w3lib.html`_)
-
- * ``scrapy.utils.multipart`` (use `urllib3`_)
-
-* The ``scrapy.utils.datatypes.MergeDict`` class is deprecated for Python 3
- code bases. Use :class:`~collections.ChainMap` instead. (:gh:`3878`)
-
-* The ``scrapy.utils.gz.is_gzipped`` function is deprecated. Use
- ``scrapy.utils.gz.gzip_magic_number`` instead.
-
-.. _urllib3: https://urllib3.readthedocs.io/en/latest/index.html
-.. _w3lib.html: https://w3lib.readthedocs.io/en/latest/w3lib.html#module-w3lib.html
-.. _w3lib.http: https://w3lib.readthedocs.io/en/latest/w3lib.html#module-w3lib.http
-
-
-Other changes
-~~~~~~~~~~~~~
-
-* It is now possible to run all tests from the same tox_ environment in
- parallel; the documentation now covers :ref:`this and other ways to run
- tests <running-tests>` (:gh:`3707`)
-
-* It is now possible to generate an API documentation coverage report
- (:gh:`3806`, :gh:`3810`, :gh:`3860`)
-
-* The :ref:`documentation policies <documentation-policies>` now require
- docstrings_ (:gh:`3701`) that follow `PEP 257`_ (:gh:`3748`)
-
-* Internal fixes and cleanup (:gh:`3629`, :gh:`3643`, :gh:`3684`,
- :gh:`3698`, :gh:`3734`, :gh:`3735`, :gh:`3736`, :gh:`3737`,
- :gh:`3809`, :gh:`3821`, :gh:`3825`, :gh:`3827`, :gh:`3833`,
- :gh:`3857`, :gh:`3877`)
-
-.. _release-1.6.0:
-
-Scrapy 1.6.0 (2019-01-30)
--------------------------
-
-Highlights:
-
-* better Windows support;
-* Python 3.7 compatibility;
-* big documentation improvements, including a switch
- from ``.extract_first()`` + ``.extract()`` API to ``.get()`` + ``.getall()``
- API;
-* feed exports, FilePipeline and MediaPipeline improvements;
-* better extensibility: :signal:`item_error` and
- :signal:`request_reached_downloader` signals; ``from_crawler`` support
- for feed exporters, feed storages and dupefilters.
-* ``scrapy.contracts`` fixes and new features;
-* telnet console security improvements, first released as a
- backport in :ref:`release-1.5.2`;
-* clean-up of the deprecated code;
-* various bug fixes, small new features and usability improvements across
- the codebase.
-
-Selector API changes
-~~~~~~~~~~~~~~~~~~~~
-
-While these are not changes in Scrapy itself, but rather in the parsel_
-library which Scrapy uses for xpath/css selectors, these changes are
-worth mentioning here. Scrapy now depends on parsel >= 1.5, and
-Scrapy documentation is updated to follow recent ``parsel`` API conventions.
-
-Most visible change is that ``.get()`` and ``.getall()`` selector
-methods are now preferred over ``.extract_first()`` and ``.extract()``.
-We feel that these new methods result in a more concise and readable code.
-See :ref:`old-extraction-api` for more details.
-
-.. note::
- There are currently **no plans** to deprecate ``.extract()``
- and ``.extract_first()`` methods.
-
-Another useful new feature is the introduction of ``Selector.attrib`` and
-``SelectorList.attrib`` properties, which make it easier to get
-attributes of HTML elements. See :ref:`selecting-attributes`.
-
-CSS selectors are cached in parsel >= 1.5, which makes them faster
-when the same CSS path is used many times. This is very common in
-case of Scrapy spiders: callbacks are usually called several times,
-on different pages.
-
-If you're using custom ``Selector`` or ``SelectorList`` subclasses,
-a **backward incompatible** change in parsel may affect your code.
-See `parsel changelog`_ for a detailed description, as well as for the
-full list of improvements.
-
-.. _parsel changelog: https://parsel.readthedocs.io/en/latest/history.html
-
-Telnet console
-~~~~~~~~~~~~~~
-
-**Backward incompatible**: Scrapy's telnet console now requires username
-and password. See :ref:`topics-telnetconsole` for more details. This change
-fixes a **security issue**; see :ref:`release-1.5.2` release notes for details.
-
-New extensibility features
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* ``from_crawler`` support is added to feed exporters and feed storages. This,
- among other things, allows to access Scrapy settings from custom feed
- storages and exporters (:gh:`1605`, :gh:`3348`).
-* ``from_crawler`` support is added to dupefilters (:gh:`2956`); this allows
- to access e.g. settings or a spider from a dupefilter.
-* :signal:`item_error` is fired when an error happens in a pipeline
- (:gh:`3256`);
-* :signal:`request_reached_downloader` is fired when Downloader gets
- a new Request; this signal can be useful e.g. for custom Schedulers
- (:gh:`3393`).
-* new SitemapSpider :meth:`~.SitemapSpider.sitemap_filter` method which allows
- to select sitemap entries based on their attributes in SitemapSpider
- subclasses (:gh:`3512`).
-* Lazy loading of Downloader Handlers is now optional; this enables better
- initialization error handling in custom Downloader Handlers (:gh:`3394`).
-
-New FilePipeline and MediaPipeline features
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`,
- :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`.
- For example, this allows to use alternative or self-hosted
- AWS-compatible providers (:gh:`2609`, :gh:`3548`).
-* ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and
- :setting:`IMAGES_STORE_GCS_ACL` (:gh:`3199`).
-
-``scrapy.contracts`` improvements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Exceptions in contracts code are handled better (:gh:`3377`);
-* ``dont_filter=True`` is used for contract requests, which allows to test
- different callbacks with the same URL (:gh:`3381`);
-* ``request_cls`` attribute in Contract subclasses allow to use different
- Request classes in contracts, for example FormRequest (:gh:`3383`).
-* Fixed errback handling in contracts, e.g. for cases where a contract
- is executed for URL which returns non-200 response (:gh:`3371`).
-
-Usability improvements
-~~~~~~~~~~~~~~~~~~~~~~
-
-* more stats for RobotsTxtMiddleware (:gh:`3100`)
-* INFO log level is used to show telnet host/port (:gh:`3115`)
-* a message is added to IgnoreRequest in RobotsTxtMiddleware (:gh:`3113`)
-* better validation of ``url`` argument in ``Response.follow`` (:gh:`3131`)
-* non-zero exit code is returned from Scrapy commands when error happens
- on spider initialization (:gh:`3226`)
-* Link extraction improvements: "ftp" is added to scheme list (:gh:`3152`);
- "flv" is added to common video extensions (:gh:`3165`)
-* better error message when an exporter is disabled (:gh:`3358`);
-* ``scrapy shell --help`` mentions syntax required for local files
- (``./file.html``) - :gh:`3496`.
-* Referer header value is added to RFPDupeFilter log messages (:gh:`3588`)
-
-Bug fixes
-~~~~~~~~~
-
-* fixed issue with extra blank lines in .csv exports under Windows
- (:gh:`3039`);
-* proper handling of pickling errors in Python 3 when serializing objects
- for disk queues (:gh:`3082`)
-* flags are now preserved when copying Requests (:gh:`3342`);
-* FormRequest.from_response clickdata shouldn't ignore elements with
- ``input[type=image]`` (:gh:`3153`).
-* FormRequest.from_response should preserve duplicate keys (:gh:`3247`)
-
-Documentation improvements
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Docs are re-written to suggest .get/.getall API instead of
- .extract/.extract_first. Also, :ref:`topics-selectors` docs are updated
- and re-structured to match latest parsel docs; they now contain more topics,
- such as :ref:`selecting-attributes` or :ref:`topics-selectors-css-extensions`
- (:gh:`3390`).
-* :ref:`topics-developer-tools` is a new tutorial which replaces
- old Firefox and Firebug tutorials (:gh:`3400`).
-* SCRAPY_PROJECT environment variable is documented (:gh:`3518`);
-* troubleshooting section is added to install instructions (:gh:`3517`);
-* improved links to beginner resources in the tutorial
- (:gh:`3367`, :gh:`3468`);
-* fixed :setting:`RETRY_HTTP_CODES` default values in docs (:gh:`3335`);
-* remove unused ``DEPTH_STATS`` option from docs (:gh:`3245`);
-* other cleanups (:gh:`3347`, :gh:`3350`, :gh:`3445`, :gh:`3544`,
- :gh:`3605`).
-
-Deprecation removals
-~~~~~~~~~~~~~~~~~~~~
-
-Compatibility shims for pre-1.0 Scrapy module names are removed
-(:gh:`3318`):
-
-* ``scrapy.command``
-* ``scrapy.contrib`` (with all submodules)
-* ``scrapy.contrib_exp`` (with all submodules)
-* ``scrapy.dupefilter``
-* ``scrapy.linkextractor``
-* ``scrapy.project``
-* ``scrapy.spider``
-* ``scrapy.spidermanager``
-* ``scrapy.squeue``
-* ``scrapy.stats``
-* ``scrapy.statscol``
-* ``scrapy.utils.decorator``
-
-See :ref:`module-relocations` for more information, or use suggestions
-from Scrapy 1.5.x deprecation warnings to update your code.
-
-Other deprecation removals:
-
-* Deprecated scrapy.interfaces.ISpiderManager is removed; please use
- scrapy.interfaces.ISpiderLoader.
-* Deprecated ``CrawlerSettings`` class is removed (:gh:`3327`).
-* Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes
- are removed (:gh:`3327`, :gh:`3359`).
-
-Other improvements, cleanups
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* All Scrapy tests now pass on Windows; Scrapy testing suite is executed
- in a Windows environment on CI (:gh:`3315`).
-* Python 3.7 support (:gh:`3326`, :gh:`3150`, :gh:`3547`).
-* Testing and CI fixes (:gh:`3526`, :gh:`3538`, :gh:`3308`,
- :gh:`3311`, :gh:`3309`, :gh:`3305`, :gh:`3210`, :gh:`3299`)
-* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name"
- optional arguments (:gh:`3231`).
-* additional files are included to sdist (:gh:`3495`);
-* code style fixes (:gh:`3405`, :gh:`3304`);
-* unneeded .strip() call is removed (:gh:`3519`);
-* collections.deque is used to store MiddlewareManager methods instead
- of a list (:gh:`3476`)
-
-.. _release-1.5.2:
-
-Scrapy 1.5.2 (2019-01-22)
--------------------------
-
-* *Security bugfix*: Telnet console extension can be easily exploited by rogue
- websites POSTing content to http://localhost:6023, we haven't found a way to
- exploit it from Scrapy, but it is very easy to trick a browser to do so and
- elevates the risk for local development environment.
-
- *The fix is backward incompatible*, it enables telnet user-password
- authentication by default with a random generated password. If you can't
- upgrade right away, please consider setting :setting:`TELNETCONSOLE_PORT`
- out of its default value.
-
- See :ref:`telnet console <topics-telnetconsole>` documentation for more info
-
-* Backport CI build failure under GCE environment due to boto import error.
-
-.. _release-1.5.1:
-
-Scrapy 1.5.1 (2018-07-12)
--------------------------
-
-This is a maintenance release with important bug fixes, but no new features:
-
-* ``O(N^2)`` gzip decompression issue which affected Python 3 and PyPy
- is fixed (:gh:`3281`);
-* skipping of TLS validation errors is improved (:gh:`3166`);
-* Ctrl-C handling is fixed in Python 3.5+ (:gh:`3096`);
-* testing fixes (:gh:`3092`, :gh:`3263`);
-* documentation improvements (:gh:`3058`, :gh:`3059`, :gh:`3089`,
- :gh:`3123`, :gh:`3127`, :gh:`3189`, :gh:`3224`, :gh:`3280`,
- :gh:`3279`, :gh:`3201`, :gh:`3260`, :gh:`3284`, :gh:`3298`,
- :gh:`3294`).
-
-
-.. _release-1.5.0:
-
-Scrapy 1.5.0 (2017-12-29)
--------------------------
-
-This release brings small new features and improvements across the codebase.
-Some highlights:
-
-* Google Cloud Storage is supported in FilesPipeline and ImagesPipeline.
-* Crawling with proxy servers becomes more efficient, as connections
- to proxies can be reused now.
-* Warnings, exception and logging messages are improved to make debugging
- easier.
-* ``scrapy parse`` command now allows to set custom request meta via
- ``--meta`` argument.
-* Compatibility with Python 3.6, PyPy and PyPy3 is improved;
- PyPy and PyPy3 are now supported officially, by running tests on CI.
-* Better default handling of HTTP 308, 522 and 524 status codes.
-* Documentation is improved, as usual.
-
-Backward Incompatible Changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* Scrapy 1.5 drops support for Python 3.3.
-* Default Scrapy User-Agent now uses https link to scrapy.org (:gh:`2983`).
- **This is technically backward-incompatible**; override
- :setting:`USER_AGENT` if you relied on old value.
-* Logging of settings overridden by ``custom_settings`` is fixed;
- **this is technically backward-incompatible** because the logger
- changes from ``[scrapy.utils.log]`` to ``[scrapy.crawler]``. If you're
- parsing Scrapy logs, please update your log parsers (:gh:`1343`).
-* LinkExtractor now ignores ``m4v`` extension by default, this is change
- in behavior.
-* 522 and 524 status codes are added to ``RETRY_HTTP_CODES`` (:gh:`2851`)
-
-New features
-~~~~~~~~~~~~
-
-- Support ``<link>`` tags in ``Response.follow`` (:gh:`2785`)
-- Support for ``ptpython`` REPL (:gh:`2654`)
-- Google Cloud Storage support for FilesPipeline and ImagesPipeline
- (:gh:`2923`).
-- New ``--meta`` option of the "scrapy parse" command allows to pass additional
- request.meta (:gh:`2883`)
-- Populate spider variable when using ``shell.inspect_response`` (:gh:`2812`)
-- Handle HTTP 308 Permanent Redirect (:gh:`2844`)
-- Add 522 and 524 to ``RETRY_HTTP_CODES`` (:gh:`2851`)
-- Log versions information at startup (:gh:`2857`)
-- ``scrapy.mail.MailSender`` now works in Python 3 (it requires Twisted 17.9.0)
-- Connections to proxy servers are reused (:gh:`2743`)
-- Add template for a downloader middleware (:gh:`2755`)
-- Explicit message for NotImplementedError when parse callback not defined
- (:gh:`2831`)
-- CrawlerProcess got an option to disable installation of root log handler
- (:gh:`2921`)
-- LinkExtractor now ignores ``m4v`` extension by default
-- Better log messages for responses over :setting:`DOWNLOAD_WARNSIZE` and
- :setting:`DOWNLOAD_MAXSIZE` limits (:gh:`2927`)
-- Show warning when a URL is put to ``Spider.allowed_domains`` instead of
- a domain (:gh:`2250`).
-
-Bug fixes
-~~~~~~~~~
-
-- Fix logging of settings overridden by ``custom_settings``;
- **this is technically backward-incompatible** because the logger
- changes from ``[scrapy.utils.log]`` to ``[scrapy.crawler]``, so please
- update your log parsers if needed (:gh:`1343`)
-- Default Scrapy User-Agent now uses https link to scrapy.org (:gh:`2983`).
- **This is technically backward-incompatible**; override
- :setting:`USER_AGENT` if you relied on old value.
-- Fix PyPy and PyPy3 test failures, support them officially
- (:gh:`2793`, :gh:`2935`, :gh:`2990`, :gh:`3050`, :gh:`2213`,
- :gh:`3048`)
-- Fix DNS resolver when ``DNSCACHE_ENABLED=False`` (:gh:`2811`)
-- Add ``cryptography`` for Debian Jessie tox test env (:gh:`2848`)
-- Add verification to check if Request callback is callable (:gh:`2766`)
-- Port ``extras/qpsclient.py`` to Python 3 (:gh:`2849`)
-- Use getfullargspec under the scenes for Python 3 to stop DeprecationWarning
- (:gh:`2862`)
-- Update deprecated test aliases (:gh:`2876`)
-- Fix ``SitemapSpider`` support for alternate links (:gh:`2853`)
-
-Docs
-~~~~
-
-- Added missing bullet point for the ``AUTOTHROTTLE_TARGET_CONCURRENCY``
- setting. (:gh:`2756`)
-- Update Contributing docs, document new support channels
- (:gh:`2762`, :gh:`3038`)
-- Include references to Scrapy subreddit in the docs
-- Fix broken links; use ``https://`` for external links
- (:gh:`2978`, :gh:`2982`, :gh:`2958`)
-- Document CloseSpider extension better (:gh:`2759`)
-- Use ``pymongo.collection.Collection.insert_one()`` in MongoDB example
- (:gh:`2781`)
-- Spelling mistake and typos
- (:gh:`2828`, :gh:`2837`, :gh:`2884`, :gh:`2924`)
-- Clarify ``CSVFeedSpider.headers`` documentation (:gh:`2826`)
-- Document ``DontCloseSpider`` exception and clarify ``spider_idle``
- (:gh:`2791`)
-- Update "Releases" section in README (:gh:`2764`)
-- Fix rst syntax in ``DOWNLOAD_FAIL_ON_DATALOSS`` docs (:gh:`2763`)
-- Small fix in description of startproject arguments (:gh:`2866`)
-- Clarify data types in Response.body docs (:gh:`2922`)
-- Add a note about ``request.meta['depth']`` to DepthMiddleware docs (:gh:`2374`)
-- Add a note about ``request.meta['dont_merge_cookies']`` to CookiesMiddleware
- docs (:gh:`2999`)
-- Up-to-date example of project structure (:gh:`2964`, :gh:`2976`)
-- A better example of ItemExporters usage (:gh:`2989`)
-- Document ``from_crawler`` methods for spider and downloader middlewares
- (:gh:`3019`)
-
-.. _release-1.4.0:
-
-Scrapy 1.4.0 (2017-05-18)
--------------------------
-
-Scrapy 1.4 does not bring that many breathtaking new features
-but quite a few handy improvements nonetheless.
-
-Scrapy now supports anonymous FTP sessions with customizable user and
-password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings.
-And if you're using Twisted version 17.1.0 or above, FTP is now available
-with Python 3.
-
-There's a new :meth:`response.follow <scrapy.http.TextResponse.follow>` method
-for creating requests; **it is now a recommended way to create Requests
-in Scrapy spiders**. This method makes it easier to write correct
-spiders; ``response.follow`` has several advantages over creating
-``scrapy.Request`` objects directly:
-
-* it handles relative URLs;
-* it works properly with non-ascii URLs on non-UTF8 pages;
-* in addition to absolute and relative URLs it supports Selectors;
- for ``<a>`` elements it can also extract their href values.
-
-For example, instead of this::
-
- for href in response.css('li.page a::attr(href)').extract():
- url = response.urljoin(href)
- yield scrapy.Request(url, self.parse, encoding=response.encoding)
-
-One can now write this::
-
- for a in response.css('li.page a'):
- yield response.follow(a, self.parse)
-
-Link extractors are also improved. They work similarly to what a regular
-modern browser would do: leading and trailing whitespace are removed
-from attributes (think ``href=" http://example.com"``) when building
-``Link`` objects. This whitespace-stripping also happens for ``action``
-attributes with ``FormRequest``.
-
-**Please also note that link extractors do not canonicalize URLs by default
-anymore.** This was puzzling users every now and then, and it's not what
-browsers do in fact, so we removed that extra transformation on extracted
-links.
-
-For those of you wanting more control on the ``Referer:`` header that Scrapy
-sends when following links, you can set your own ``Referrer Policy``.
-Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and
-blindly set it to the URL of the response that generated the HTTP request
-(which could leak information on your URL seeds).
-By default, Scrapy now behaves much like your regular browser does.
-And this policy is fully customizable with W3C standard values
-(or with something really custom of your own if you wish).
-See :setting:`REFERRER_POLICY` for details.
-
-To make Scrapy spiders easier to debug, Scrapy logs more stats by default
-in 1.4: memory usage stats, detailed retry stats, detailed HTTP error code
-stats. A similar change is that HTTP cache path is also visible in logs now.
-
-Last but not least, Scrapy now has the option to make JSON and XML items
-more human-readable, with newlines between items and even custom indenting
-offset, using the new :setting:`FEED_EXPORT_INDENT` setting.
-
-Enjoy! (Or read on for the rest of changes in this release.)
-
-Deprecations and Backward Incompatible Changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Default to ``canonicalize=False`` in
- :class:`scrapy.linkextractors.LinkExtractor
- <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
- (:gh:`2537`, fixes :gh:`1941` and :gh:`1982`):
- **warning, this is technically backward-incompatible**
-- Enable memusage extension by default (:gh:`2539`, fixes :gh:`2187`);
- **this is technically backward-incompatible** so please check if you have
- any non-default ``MEMUSAGE_***`` options set.
-- ``EDITOR`` environment variable now takes precedence over ``EDITOR``
- option defined in settings.py (:gh:`1829`); Scrapy default settings
- no longer depend on environment variables. **This is technically a backward
- incompatible change**.
-- ``Spider.make_requests_from_url`` is deprecated
- (:gh:`1728`, fixes :gh:`1495`).
-
-New Features
-~~~~~~~~~~~~
-
-- Accept proxy credentials in :reqmeta:`proxy` request meta key (:gh:`2526`)
-- Support `brotli-compressed`_ content; requires optional `brotlipy`_
- (:gh:`2535`)
-- New :ref:`response.follow <response-follow-example>` shortcut
- for creating requests (:gh:`1940`)
-- Added ``flags`` argument and attribute to :class:`~scrapy.Request`
- objects (:gh:`2047`)
-- Support Anonymous FTP (:gh:`2342`)
-- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/<reason>``
- stats to :class:`RetryMiddleware <scrapy.downloadermiddlewares.retry.RetryMiddleware>`
- (:gh:`2543`)
-- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/<status>``
- stats to :class:`HttpErrorMiddleware <scrapy.spidermiddlewares.httperror.HttpErrorMiddleware>`
- (:gh:`2566`)
-- Customizable :setting:`Referrer policy <REFERRER_POLICY>` in
- :class:`RefererMiddleware <scrapy.spidermiddlewares.referer.RefererMiddleware>`
- (:gh:`2306`)
-- New ``data:`` URI download handler (:gh:`2334`, fixes :gh:`2156`)
-- Log cache directory when HTTP Cache is used (:gh:`2611`, fixes :gh:`2604`)
-- Warn users when project contains duplicate spider names (fixes :gh:`2181`)
-- ``scrapy.utils.datatypes.CaselessDict`` now accepts ``Mapping`` instances and
- not only dicts (:gh:`2646`)
-- :ref:`Media downloads <topics-media-pipeline>`, with
- :class:`~scrapy.pipelines.files.FilesPipeline` or
- :class:`~scrapy.pipelines.images.ImagesPipeline`, can now optionally handle
- HTTP redirects using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting
- (:gh:`2616`, fixes :gh:`2004`)
-- Accept non-complete responses from websites using a new
- :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:gh:`2590`, fixes :gh:`2586`)
-- Optional pretty-printing of JSON and XML items via
- :setting:`FEED_EXPORT_INDENT` setting (:gh:`2456`, fixes :gh:`1327`)
-- Allow dropping fields in ``FormRequest.from_response`` formdata when
- ``None`` value is passed (:gh:`667`)
-- Per-request retry times with the new :reqmeta:`max_retry_times` meta key
- (:gh:`2642`)
-- ``python -m scrapy`` as a more explicit alternative to ``scrapy`` command
- (:gh:`2740`)
-
-.. _brotli-compressed: https://www.ietf.org/rfc/rfc7932.txt
-.. _brotlipy: https://github.com/python-hyper/brotlipy/
-
-Bug fixes
-~~~~~~~~~
-
-- LinkExtractor now strips leading and trailing whitespaces from attributes
- (:gh:`2547`, fixes :gh:`1614`)
-- Properly handle whitespaces in action attribute in
- :class:`~scrapy.FormRequest` (:gh:`2548`)
-- Buffer CONNECT response bytes from proxy until all HTTP headers are received
- (:gh:`2495`, fixes :gh:`2491`)
-- FTP downloader now works on Python 3, provided you use Twisted>=17.1
- (:gh:`2599`)
-- Use body to choose response type after decompressing content (:gh:`2393`,
- fixes :gh:`2145`)
-- Always decompress ``Content-Encoding: gzip`` at :class:`HttpCompressionMiddleware
- <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>` stage (:gh:`2391`)
-- Respect custom log level in ``Spider.custom_settings`` (:gh:`2581`,
- fixes :gh:`1612`)
-- 'make htmlview' fix for macOS (:gh:`2661`)
-- Remove "commands" from the command list (:gh:`2695`)
-- Fix duplicate Content-Length header for POST requests with empty body (:gh:`2677`)
-- Properly cancel large downloads, i.e. above :setting:`DOWNLOAD_MAXSIZE` (:gh:`1616`)
-- ImagesPipeline: fixed processing of transparent PNG images with palette
- (:gh:`2675`)
-
-Cleanups & Refactoring
-~~~~~~~~~~~~~~~~~~~~~~
-
-- Tests: remove temp files and folders (:gh:`2570`),
- fixed ProjectUtilsTest on macOS (:gh:`2569`),
- use portable pypy for Linux on Travis CI (:gh:`2710`)
-- Separate building request from ``_requests_to_follow`` in CrawlSpider (:gh:`2562`)
-- Remove “Python 3 progress” badge (:gh:`2567`)
-- Add a couple more lines to ``.gitignore`` (:gh:`2557`)
-- Remove bumpversion prerelease configuration (:gh:`2159`)
-- Add codecov.yml file (:gh:`2750`)
-- Set context factory implementation based on Twisted version (:gh:`2577`,
- fixes :gh:`2560`)
-- Add omitted ``self`` arguments in default project middleware template (:gh:`2595`)
-- Remove redundant ``slot.add_request()`` call in ExecutionEngine (:gh:`2617`)
-- Catch more specific ``os.error`` exception in
- ``scrapy.pipelines.files.FSFilesStore`` (:gh:`2644`)
-- Change "localhost" test server certificate (:gh:`2720`)
-- Remove unused ``MEMUSAGE_REPORT`` setting (:gh:`2576`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Binary mode is required for exporters (:gh:`2564`, fixes :gh:`2553`)
-- Mention issue with :meth:`.FormRequest.from_response` due to bug in lxml (:gh:`2572`)
-- Use single quotes uniformly in templates (:gh:`2596`)
-- Document :reqmeta:`ftp_user` and :reqmeta:`ftp_password` meta keys (:gh:`2587`)
-- Removed section on deprecated ``contrib/`` (:gh:`2636`)
-- Recommend Anaconda when installing Scrapy on Windows
- (:gh:`2477`, fixes :gh:`2475`)
-- FAQ: rewrite note on Python 3 support on Windows (:gh:`2690`)
-- Rearrange selector sections (:gh:`2705`)
-- Remove ``__nonzero__`` from :class:`~scrapy.selector.SelectorList`
- docs (:gh:`2683`)
-- Mention how to disable request filtering in documentation of
- :setting:`DUPEFILTER_CLASS` setting (:gh:`2714`)
-- Add sphinx_rtd_theme to docs setup readme (:gh:`2668`)
-- Open file in text mode in JSON item writer example (:gh:`2729`)
-- Clarify ``allowed_domains`` example (:gh:`2670`)
-
-
-.. _release-1.3.3:
-
-Scrapy 1.3.3 (2017-03-10)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Make ``SpiderLoader`` raise ``ImportError`` again by default for missing
- dependencies and wrong :setting:`SPIDER_MODULES`.
- These exceptions were silenced as warnings since 1.3.0.
- A new setting is introduced to toggle between warning or exception if needed ;
- see :setting:`SPIDER_LOADER_WARN_ONLY` for details.
-
-.. _release-1.3.2:
-
-Scrapy 1.3.2 (2017-02-13)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Preserve request class when converting to/from dicts (utils.reqser) (:gh:`2510`).
-- Use consistent selectors for author field in tutorial (:gh:`2551`).
-- Fix TLS compatibility in Twisted 17+ (:gh:`2558`)
-
-.. _release-1.3.1:
-
-Scrapy 1.3.1 (2017-02-08)
--------------------------
-
-New features
-~~~~~~~~~~~~
-
-- Support ``'True'`` and ``'False'`` string values for boolean settings (:gh:`2519`);
- you can now do something like ``scrapy crawl myspider -s REDIRECT_ENABLED=False``.
-- Support kwargs with ``response.xpath()`` to use :ref:`XPath variables <topics-selectors-xpath-variables>`
- and ad-hoc namespaces declarations ;
- this requires at least Parsel v1.1 (:gh:`2457`).
-- Add support for Python 3.6 (:gh:`2485`).
-- Run tests on PyPy (warning: some tests still fail, so PyPy is not supported yet).
-
-Bug fixes
-~~~~~~~~~
-
-- Enforce ``DNS_TIMEOUT`` setting (:gh:`2496`).
-- Fix :command:`view` command ; it was a regression in v1.3.0 (:gh:`2503`).
-- Fix tests regarding ``*_EXPIRES settings`` with Files/Images pipelines (:gh:`2460`).
-- Fix name of generated pipeline class when using basic project template (:gh:`2466`).
-- Fix compatibility with Twisted 17+ (:gh:`2496`, :gh:`2528`).
-- Fix ``scrapy.Item`` inheritance on Python 3.6 (:gh:`2511`).
-- Enforce numeric values for components order in ``SPIDER_MIDDLEWARES``,
- ``DOWNLOADER_MIDDLEWARES``, ``EXTENSIONS`` and ``SPIDER_CONTRACTS`` (:gh:`2420`).
-
-Documentation
-~~~~~~~~~~~~~
-
-- Reword Code of Conduct section and upgrade to Contributor Covenant v1.4
- (:gh:`2469`).
-- Clarify that passing spider arguments converts them to spider attributes
- (:gh:`2483`).
-- Document ``formid`` argument on ``FormRequest.from_response()`` (:gh:`2497`).
-- Add .rst extension to README files (:gh:`2507`).
-- Mention LevelDB cache storage backend (:gh:`2525`).
-- Use ``yield`` in sample callback code (:gh:`2533`).
-- Add note about HTML entities decoding with ``.re()/.re_first()`` (:gh:`1704`).
-- Typos (:gh:`2512`, :gh:`2534`, :gh:`2531`).
-
-Cleanups
-~~~~~~~~
-
-- Remove redundant check in ``MetaRefreshMiddleware`` (:gh:`2542`).
-- Faster checks in ``LinkExtractor`` for allow/deny patterns (:gh:`2538`).
-- Remove dead code supporting old Twisted versions (:gh:`2544`).
-
-
-.. _release-1.3.0:
-
-Scrapy 1.3.0 (2016-12-21)
--------------------------
-
-This release comes rather soon after 1.2.2 for one main reason:
-it was found out that releases since 0.18 up to 1.2.2 (included) use
-some backported code from Twisted (``scrapy.xlib.tx.*``),
-even if newer Twisted modules are available.
-Scrapy now uses ``twisted.web.client`` and ``twisted.internet.endpoints`` directly.
-(See also cleanups below.)
-
-As it is a major change, we wanted to get the bug fix out quickly
-while not breaking any projects using the 1.2 series.
-
-New Features
-~~~~~~~~~~~~
-
-- ``MailSender`` now accepts single strings as values for ``to`` and ``cc``
- arguments (:gh:`2272`)
-- ``scrapy fetch url``, ``scrapy shell url`` and ``fetch(url)`` inside
- Scrapy shell now follow HTTP redirections by default (:gh:`2290`);
- See :command:`fetch` and :command:`shell` for details.
-- ``HttpErrorMiddleware`` now logs errors with ``INFO`` level instead of ``DEBUG``;
- this is technically **backward incompatible** so please check your log parsers.
-- By default, logger names now use a long-form path, e.g. ``[scrapy.extensions.logstats]``,
- instead of the shorter "top-level" variant of prior releases (e.g. ``[scrapy]``);
- this is **backward incompatible** if you have log parsers expecting the short
- logger name part. You can switch back to short logger names using :setting:`LOG_SHORT_NAMES`
- set to ``True``.
-
-Dependencies & Cleanups
-~~~~~~~~~~~~~~~~~~~~~~~
-
-- Scrapy now requires Twisted >= 13.1 which is the case for many Linux
- distributions already.
-- As a consequence, we got rid of ``scrapy.xlib.tx.*`` modules, which
- copied some of Twisted code for users stuck with an "old" Twisted version
-- ``ChunkedTransferMiddleware`` is deprecated and removed from the default
- downloader middlewares.
-
-.. _release-1.2.3:
-
-Scrapy 1.2.3 (2017-03-03)
--------------------------
-
-- Packaging fix: disallow unsupported Twisted versions in setup.py
-
-
-.. _release-1.2.2:
-
-Scrapy 1.2.2 (2016-12-06)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Fix a cryptic traceback when a pipeline fails on ``open_spider()`` (:gh:`2011`)
-- Fix embedded IPython shell variables (fixing :gh:`396` that re-appeared
- in 1.2.0, fixed in :gh:`2418`)
-- A couple of patches when dealing with robots.txt:
-
- - handle (non-standard) relative sitemap URLs (:gh:`2390`)
- - handle non-ASCII URLs and User-Agents in Python 2 (:gh:`2373`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- Document ``"download_latency"`` key in ``Request``'s ``meta`` dict (:gh:`2033`)
-- Remove page on (deprecated & unsupported) Ubuntu packages from ToC (:gh:`2335`)
-- A few fixed typos (:gh:`2346`, :gh:`2369`, :gh:`2369`, :gh:`2380`)
- and clarifications (:gh:`2354`, :gh:`2325`, :gh:`2414`)
-
-Other changes
-~~~~~~~~~~~~~
-
-- Advertize `conda-forge`_ as Scrapy's official conda channel (:gh:`2387`)
-- More helpful error messages when trying to use ``.css()`` or ``.xpath()``
- on non-Text Responses (:gh:`2264`)
-- ``startproject`` command now generates a sample ``middlewares.py`` file (:gh:`2335`)
-- Add more dependencies' version info in ``scrapy version`` verbose output (:gh:`2404`)
-- Remove all ``*.pyc`` files from source distribution (:gh:`2386`)
-
-.. _conda-forge: https://anaconda.org/conda-forge/scrapy
-
-
-.. _release-1.2.1:
-
-Scrapy 1.2.1 (2016-10-21)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Include OpenSSL's more permissive default ciphers when establishing
- TLS/SSL connections (:gh:`2314`).
-- Fix "Location" HTTP header decoding on non-ASCII URL redirects (:gh:`2321`).
-
-Documentation
-~~~~~~~~~~~~~
-
-- Fix JsonWriterPipeline example (:gh:`2302`).
-- Various notes: :gh:`2330` on spider names,
- :gh:`2329` on middleware methods processing order,
- :gh:`2327` on getting multi-valued HTTP headers as lists.
-
-Other changes
-~~~~~~~~~~~~~
-
-- Removed ``www.`` from ``start_urls`` in built-in spider templates (:gh:`2299`).
-
-
-.. _release-1.2.0:
-
-Scrapy 1.2.0 (2016-10-03)
--------------------------
-
-New Features
-~~~~~~~~~~~~
-
-- New :setting:`FEED_EXPORT_ENCODING` setting to customize the encoding
- used when writing items to a file.
- This can be used to turn off ``\uXXXX`` escapes in JSON output.
- This is also useful for those wanting something else than UTF-8
- for XML or CSV output (:gh:`2034`).
-- ``startproject`` command now supports an optional destination directory
- to override the default one based on the project name (:gh:`2005`).
-- New :setting:`SCHEDULER_DEBUG` setting to log requests serialization
- failures (:gh:`1610`).
-- JSON encoder now supports serialization of ``set`` instances (:gh:`2058`).
-- Interpret ``application/json-amazonui-streaming`` as ``TextResponse`` (:gh:`1503`).
-- ``scrapy`` is imported by default when using shell tools (:command:`shell`,
- :ref:`inspect_response <topics-shell-inspect-response>`) (:gh:`2248`).
-
-Bug fixes
-~~~~~~~~~
-
-- DefaultRequestHeaders middleware now runs before UserAgent middleware
- (:gh:`2088`). **Warning: this is technically backward incompatible**,
- though we consider this a bug fix.
-- HTTP cache extension and plugins that use the ``.scrapy`` data directory now
- work outside projects (:gh:`1581`). **Warning: this is technically
- backward incompatible**, though we consider this a bug fix.
-- ``Selector`` does not allow passing both ``response`` and ``text`` anymore
- (:gh:`2153`).
-- Fixed logging of wrong callback name with ``scrapy parse`` (:gh:`2169`).
-- Fix for an odd gzip decompression bug (:gh:`1606`).
-- Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse <parse>`
- (:gh:`2225`).
-- Fix for invalid JSON and XML files when spider yields no items (:gh:`872`).
-- Implement ``flush()`` for ``StreamLogger`` avoiding a warning in logs (:gh:`2125`).
-
-Refactoring
-~~~~~~~~~~~
-
-- ``canonicalize_url`` has been moved to `w3lib.url`_ (:gh:`2168`).
-
-.. _w3lib.url: https://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.url.canonicalize_url
-
-Tests & Requirements
-~~~~~~~~~~~~~~~~~~~~
-
-Scrapy's new requirements baseline is Debian 8 "Jessie". It was previously
-Ubuntu 12.04 Precise.
-What this means in practice is that we run continuous integration tests
-with these (main) packages versions at a minimum:
-Twisted 14.0, pyOpenSSL 0.14, lxml 3.4.
-
-Scrapy may very well work with older versions of these packages
-(the code base still has switches for older Twisted versions for example)
-but it is not guaranteed (because it's not tested anymore).
-
-Documentation
-~~~~~~~~~~~~~
-
-- Grammar fixes: :gh:`2128`, :gh:`1566`.
-- Download stats badge removed from README (:gh:`2160`).
-- New Scrapy :ref:`architecture diagram <topics-architecture>` (:gh:`2165`).
-- Updated ``Response`` parameters documentation (:gh:`2197`).
-- Reworded misleading ``RANDOMIZE_DOWNLOAD_DELAY`` description (:gh:`2190`).
-- Add StackOverflow as a support channel (:gh:`2257`).
-
-.. _release-1.1.4:
-
-Scrapy 1.1.4 (2017-03-03)
--------------------------
-
-- Packaging fix: disallow unsupported Twisted versions in setup.py
-
-.. _release-1.1.3:
-
-Scrapy 1.1.3 (2016-09-22)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Class attributes for subclasses of ``ImagesPipeline`` and ``FilesPipeline``
- work as they did before 1.1.1 (:gh:`2243`, fixes :gh:`2198`)
-
-Documentation
-~~~~~~~~~~~~~
-
-- :ref:`Overview <intro-overview>` and :ref:`tutorial <intro-tutorial>`
- rewritten to use http://toscrape.com websites
- (:gh:`2236`, :gh:`2249`, :gh:`2252`).
-
-.. _release-1.1.2:
-
-Scrapy 1.1.2 (2016-08-18)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Introduce a missing :setting:`IMAGES_STORE_S3_ACL` setting to override
- the default ACL policy in ``ImagesPipeline`` when uploading images to S3
- (note that default ACL policy is "private" -- instead of "public-read" --
- since Scrapy 1.1.0)
-- :setting:`IMAGES_EXPIRES` default value set back to 90
- (the regression was introduced in 1.1.1)
-
-.. _release-1.1.1:
-
-Scrapy 1.1.1 (2016-07-13)
--------------------------
-
-Bug fixes
-~~~~~~~~~
-
-- Add "Host" header in CONNECT requests to HTTPS proxies (:gh:`2069`)
-- Use response ``body`` when choosing response class
- (:gh:`2001`, fixes :gh:`2000`)
-- Do not fail on canonicalizing URLs with wrong netlocs
- (:gh:`2038`, fixes :gh:`2010`)
-- a few fixes for ``HttpCompressionMiddleware`` (and ``SitemapSpider``):
-
- - Do not decode HEAD responses (:gh:`2008`, fixes :gh:`1899`)
- - Handle charset parameter in gzip Content-Type header
- (:gh:`2050`, fixes :gh:`2049`)
- - Do not decompress gzip octet-stream responses
- (:gh:`2065`, fixes :gh:`2063`)
-
-- Catch (and ignore with a warning) exception when verifying certificate
- against IP-address hosts (:gh:`2094`, fixes :gh:`2092`)
-- Make ``FilesPipeline`` and ``ImagesPipeline`` backward compatible again
- regarding the use of legacy class attributes for customization
- (:gh:`1989`, fixes :gh:`1985`)
-
-
-New features
-~~~~~~~~~~~~
-
-- Enable genspider command outside project folder (:gh:`2052`)
-- Retry HTTPS CONNECT ``TunnelError`` by default (:gh:`1974`)
-
-
-Documentation
-~~~~~~~~~~~~~
-
-- ``FEED_TEMPDIR`` setting at lexicographical position (:commit:`9b3c72c`)
-- Use idiomatic ``.extract_first()`` in overview (:gh:`1994`)
-- Update years in copyright notice (:commit:`c2c8036`)
-- Add information and example on errbacks (:gh:`1995`)
-- Use "url" variable in downloader middleware example (:gh:`2015`)
-- Grammar fixes (:gh:`2054`, :gh:`2120`)
-- New FAQ entry on using BeautifulSoup in spider callbacks (:gh:`2048`)
-- Add notes about Scrapy not working on Windows with Python 3 (:gh:`2060`)
-- Encourage complete titles in pull requests (:gh:`2026`)
-
-Tests
-~~~~~
-
-- Upgrade py.test requirement on Travis CI and Pin pytest-cov to 2.2.1 (:gh:`2095`)
-
-.. _release-1.1.0:
-
-Scrapy 1.1.0 (2016-05-11)
--------------------------
-
-This 1.1 release brings a lot of interesting features and bug fixes:
-
-- Scrapy 1.1 has beta Python 3 support (requires Twisted >= 15.5). See
- :ref:`news_betapy3` for more details and some limitations.
-- Hot new features:
-
- - Item loaders now support nested loaders (:gh:`1467`).
- - ``FormRequest.from_response`` improvements (:gh:`1382`, :gh:`1137`).
- - Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improved
- AutoThrottle docs (:gh:`1324`).
- - Added ``response.text`` to get body as unicode (:gh:`1730`).
- - Anonymous S3 connections (:gh:`1358`).
- - Deferreds in downloader middlewares (:gh:`1473`). This enables better
- robots.txt handling (:gh:`1471`).
- - HTTP caching now follows RFC2616 more closely, added settings
- :setting:`HTTPCACHE_ALWAYS_STORE` and
- :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:gh:`1151`).
- - Selectors were extracted to the parsel_ library (:gh:`1409`). This means
- you can use Scrapy Selectors without Scrapy and also upgrade the
- selectors engine without needing to upgrade Scrapy.
- - HTTPS downloader now does TLS protocol negotiation by default,
- instead of forcing TLS 1.0. You can also set the SSL/TLS method
- using the new ``DOWNLOADER_CLIENT_TLS_METHOD`` setting.
-
-- These bug fixes may require your attention:
-
- - Don't retry bad requests (HTTP 400) by default (:gh:`1289`).
- If you need the old behavior, add ``400`` to :setting:`RETRY_HTTP_CODES`.
- - Fix shell files argument handling (:gh:`1710`, :gh:`1550`).
- If you try ``scrapy shell index.html`` it will try to load the URL ``http://index.html``,
- use ``scrapy shell ./index.html`` to load a local file.
- - Robots.txt compliance is now enabled by default for newly-created projects
- (:gh:`1724`). Scrapy will also wait for robots.txt to be downloaded
- before proceeding with the crawl (:gh:`1735`). If you want to disable
- this behavior, update :setting:`ROBOTSTXT_OBEY` in ``settings.py`` file
- after creating a new project.
- - Exporters now work on unicode, instead of bytes by default (:gh:`1080`).
- If you use :class:`~scrapy.exporters.PythonItemExporter`, you may want to
- update your code to disable binary mode which is now deprecated.
- - Accept XML node names containing dots as valid (:gh:`1533`).
- - When uploading files or images to S3 (with ``FilesPipeline`` or
- ``ImagesPipeline``), the default ACL policy is now "private" instead
- of "public" **Warning: backward incompatible!**.
- You can use :setting:`FILES_STORE_S3_ACL` to change it.
- - We've reimplemented ``canonicalize_url()`` for more correct output,
- especially for URLs with non-ASCII characters (:gh:`1947`).
- This could change link extractors output compared to previous Scrapy versions.
- This may also invalidate some cache entries you could still have from pre-1.1 runs.
- **Warning: backward incompatible!**.
-
-Keep reading for more details on other improvements and bug fixes.
-
-.. _news_betapy3:
-
-Beta Python 3 Support
-~~~~~~~~~~~~~~~~~~~~~
-
-We have been hard at work to make Scrapy run on Python 3. As a result, now
-you can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some
-features are still missing (and some may never be ported).
-
-
-Almost all builtin extensions/middlewares are expected to work.
-However, we are aware of some limitations in Python 3:
-
-- Scrapy does not work on Windows with Python 3
-- Sending emails is not supported
-- FTP download handler is not supported
-- Telnet console is not supported
-
-Additional New Features and Enhancements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Scrapy now has a `Code of Conduct`_ (:gh:`1681`).
-- Command line tool now has completion for zsh (:gh:`934`).
-- Improvements to ``scrapy shell``:
-
- - Support for bpython and configure preferred Python shell via
- ``SCRAPY_PYTHON_SHELL`` (:gh:`1100`, :gh:`1444`).
- - Support URLs without scheme (:gh:`1498`)
- **Warning: backward incompatible!**
- - Bring back support for relative file path (:gh:`1710`, :gh:`1550`).
-
-- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check
- interval (:gh:`1282`).
-- Download handlers are now lazy-loaded on first request using their
- scheme (:gh:`1390`, :gh:`1421`).
-- HTTPS download handlers do not force TLS 1.0 anymore; instead,
- OpenSSL's ``SSLv23_method()/TLS_method()`` is used allowing to try
- negotiating with the remote hosts the highest TLS protocol version
- it can (:gh:`1794`, :gh:`1629`).
-- ``RedirectMiddleware`` now skips the status codes from
- ``handle_httpstatus_list`` on spider attribute
- or in ``Request``'s ``meta`` key (:gh:`1334`, :gh:`1364`,
- :gh:`1447`).
-- Form submission:
-
- - now works with ``<button>`` elements too (:gh:`1469`).
- - an empty string is now used for submit buttons without a value
- (:gh:`1472`)
-
-- Dict-like settings now have per-key priorities
- (:gh:`1135`, :gh:`1149` and :gh:`1586`).
-- Sending non-ASCII emails (:gh:`1662`)
-- ``CloseSpider`` and ``SpiderState`` extensions now get disabled if no relevant
- setting is set (:gh:`1723`, :gh:`1725`).
-- Added method ``ExecutionEngine.close`` (:gh:`1423`).
-- Added method ``CrawlerRunner.create_crawler`` (:gh:`1528`).
-- Scheduler priority queue can now be customized via
- :setting:`SCHEDULER_PRIORITY_QUEUE` (:gh:`1822`).
-- ``.pps`` links are now ignored by default in link extractors (:gh:`1835`).
-- temporary data folder for FTP and S3 feed storages can be customized
- using a new :setting:`FEED_TEMPDIR` setting (:gh:`1847`).
-- ``FilesPipeline`` and ``ImagesPipeline`` settings are now instance attributes
- instead of class attributes, enabling spider-specific behaviors (:gh:`1891`).
-- ``JsonItemExporter`` now formats opening and closing square brackets
- on their own line (first and last lines of output file) (:gh:`1950`).
-- If available, ``botocore`` is used for ``S3FeedStorage``, ``S3DownloadHandler``
- and ``S3FilesStore`` (:gh:`1761`, :gh:`1883`).
-- Tons of documentation updates and related fixes (:gh:`1291`, :gh:`1302`,
- :gh:`1335`, :gh:`1683`, :gh:`1660`, :gh:`1642`, :gh:`1721`,
- :gh:`1727`, :gh:`1879`).
-- Other refactoring, optimizations and cleanup (:gh:`1476`, :gh:`1481`,
- :gh:`1477`, :gh:`1315`, :gh:`1290`, :gh:`1750`, :gh:`1881`).
-
-.. _Code of Conduct: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
-
-
-Deprecations and Removals
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Added ``to_bytes`` and ``to_unicode``, deprecated ``str_to_unicode`` and
- ``unicode_to_str`` functions (:gh:`778`).
-- ``binary_is_text`` is introduced, to replace use of ``isbinarytext``
- (but with inverse return value) (:gh:`1851`)
-- The ``optional_features`` set has been removed (:gh:`1359`).
-- The ``--lsprof`` command line option has been removed (:gh:`1689`).
- **Warning: backward incompatible**, but doesn't break user code.
-- The following datatypes were deprecated (:gh:`1720`):
-
- + ``scrapy.utils.datatypes.MultiValueDictKeyError``
- + ``scrapy.utils.datatypes.MultiValueDict``
- + ``scrapy.utils.datatypes.SiteNode``
-
-- The previously bundled ``scrapy.xlib.pydispatch`` library was deprecated and
- replaced by `pydispatcher <https://pypi.org/project/PyDispatcher/>`_.
-
-
-Relocations
-~~~~~~~~~~~
-
-- ``telnetconsole`` was relocated to ``extensions/`` (:gh:`1524`).
-
- + Note: telnet is not enabled on Python 3
- (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)
-
-
-Bugfixes
-~~~~~~~~
-
-- Scrapy does not retry requests that got a ``HTTP 400 Bad Request``
- response anymore (:gh:`1289`). **Warning: backward incompatible!**
-- Support empty password for http_proxy config (:gh:`1274`).
-- Interpret ``application/x-json`` as ``TextResponse`` (:gh:`1333`).
-- Support link rel attribute with multiple values (:gh:`1201`).
-- Fixed ``scrapy.FormRequest.from_response`` when there is a ``<base>``
- tag (:gh:`1564`).
-- Fixed :setting:`TEMPLATES_DIR` handling (:gh:`1575`).
-- Various ``FormRequest`` fixes (:gh:`1595`, :gh:`1596`, :gh:`1597`).
-- Makes ``_monkeypatches`` more robust (:gh:`1634`).
-- Fixed bug on ``XMLItemExporter`` with non-string fields in
- items (:gh:`1738`).
-- Fixed startproject command in macOS (:gh:`1635`).
-- Fixed :class:`~scrapy.exporters.PythonItemExporter` and CSVExporter for
- non-string item types (:gh:`1737`).
-- Various logging related fixes (:gh:`1294`, :gh:`1419`, :gh:`1263`,
- :gh:`1624`, :gh:`1654`, :gh:`1722`, :gh:`1726` and :gh:`1303`).
-- Fixed bug in ``utils.template.render_templatefile()`` (:gh:`1212`).
-- sitemaps extraction from ``robots.txt`` is now case-insensitive (:gh:`1902`).
-- HTTPS+CONNECT tunnels could get mixed up when using multiple proxies
- to same remote host (:gh:`1912`).
-
-.. _release-1.0.7:
-
-Scrapy 1.0.7 (2017-03-03)
--------------------------
-
-- Packaging fix: disallow unsupported Twisted versions in setup.py
-
-.. _release-1.0.6:
-
-Scrapy 1.0.6 (2016-05-04)
--------------------------
-
-- FIX: RetryMiddleware is now robust to non-standard HTTP status codes (:gh:`1857`)
-- FIX: Filestorage HTTP cache was checking wrong modified time (:gh:`1875`)
-- DOC: Support for Sphinx 1.4+ (:gh:`1893`)
-- DOC: Consistency in selectors examples (:gh:`1869`)
-
-.. _release-1.0.5:
-
-Scrapy 1.0.5 (2016-02-04)
--------------------------
-
-- FIX: [Backport] Ignore bogus links in LinkExtractors (fixes :gh:`907`, :commit:`108195e`)
-- TST: Changed buildbot makefile to use 'pytest' (:commit:`1f3d90a`)
-- DOC: Fixed typos in tutorial and media-pipeline (:commit:`808a9ea` and :commit:`803bd87`)
-- DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`)
-
-.. _release-1.0.4:
-
-Scrapy 1.0.4 (2015-12-30)
--------------------------
-
-- Ignoring xlib/tx folder, depending on Twisted version. (:commit:`7dfa979`)
-- Run on new travis-ci infra (:commit:`6e42f0b`)
-- Spelling fixes (:commit:`823a1cc`)
-- escape nodename in xmliter regex (:commit:`da3c155`)
-- test xml nodename with dots (:commit:`4418fc3`)
-- TST don't use broken Pillow version in tests (:commit:`a55078c`)
-- disable log on version command. closes #1426 (:commit:`86fc330`)
-- disable log on startproject command (:commit:`db4c9fe`)
-- Add PyPI download stats badge (:commit:`df2b944`)
-- don't run tests twice on Travis if a PR is made from a scrapy/scrapy branch (:commit:`a83ab41`)
-- Add Python 3 porting status badge to the README (:commit:`73ac80d`)
-- fixed RFPDupeFilter persistence (:commit:`97d080e`)
-- TST a test to show that dupefilter persistence is not working (:commit:`97f2fb3`)
-- explicit close file on file:// scheme handler (:commit:`d9b4850`)
-- Disable dupefilter in shell (:commit:`c0d0734`)
-- DOC: Add captions to toctrees which appear in sidebar (:commit:`aa239ad`)
-- DOC Removed pywin32 from install instructions as it's already declared as dependency. (:commit:`10eb400`)
-- Added installation notes about using Conda for Windows and other OSes. (:commit:`1c3600a`)
-- Fixed minor grammar issues. (:commit:`7f4ddd5`)
-- fixed a typo in the documentation. (:commit:`b71f677`)
-- Version 1 now exists (:commit:`5456c0e`)
-- fix another invalid xpath error (:commit:`0a1366e`)
-- fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst (:commit:`ca8d60f`)
-- Typos corrections (:commit:`7067117`)
-- fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware (:commit:`32f115c`)
-- Add note to Ubuntu install section about Debian compatibility (:commit:`23fda69`)
-- Replace alternative macOS install workaround with virtualenv (:commit:`98b63ee`)
-- Reference Homebrew's homepage for installation instructions (:commit:`1925db1`)
-- Add oldest supported tox version to contributing docs (:commit:`5d10d6d`)
-- Note in install docs about pip being already included in python>=2.7.9 (:commit:`85c980e`)
-- Add non-python dependencies to Ubuntu install section in the docs (:commit:`fbd010d`)
-- Add macOS installation section to docs (:commit:`d8f4cba`)
-- DOC(ENH): specify path to rtd theme explicitly (:commit:`de73b1a`)
-- minor: scrapy.Spider docs grammar (:commit:`1ddcc7b`)
-- Make common practices sample code match the comments (:commit:`1b85bcf`)
-- nextcall repetitive calls (heartbeats). (:commit:`55f7104`)
-- Backport fix compatibility with Twisted 15.4.0 (:commit:`b262411`)
-- pin pytest to 2.7.3 (:commit:`a6535c2`)
-- Merge pull request #1512 from mgedmin/patch-1 (:commit:`8876111`)
-- Merge pull request #1513 from mgedmin/patch-2 (:commit:`5d4daf8`)
-- Typo (:commit:`f8d0682`)
-- Fix list formatting (:commit:`5f83a93`)
-- fix Scrapy squeue tests after recent changes to queuelib (:commit:`3365c01`)
-- Merge pull request #1475 from rweindl/patch-1 (:commit:`2d688cd`)
-- Update tutorial.rst (:commit:`fbc1f25`)
-- Merge pull request #1449 from rhoekman/patch-1 (:commit:`7d6538c`)
-- Small grammatical change (:commit:`8752294`)
-- Add openssl version to version command (:commit:`13c45ac`)
-
-.. _release-1.0.3:
-
-Scrapy 1.0.3 (2015-08-11)
--------------------------
-
-- add service_identity to Scrapy install_requires (:commit:`cbc2501`)
-- Workaround for travis#296 (:commit:`66af9cd`)
-
-.. _release-1.0.2:
-
-Scrapy 1.0.2 (2015-08-06)
--------------------------
-
-- Twisted 15.3.0 does not raises PicklingError serializing lambda functions (:commit:`b04dd7d`)
-- Minor method name fix (:commit:`6f85c7f`)
-- minor: scrapy.Spider grammar and clarity (:commit:`9c9d2e0`)
-- Put a blurb about support channels in CONTRIBUTING (:commit:`c63882b`)
-- Fixed typos (:commit:`a9ae7b0`)
-- Fix doc reference. (:commit:`7c8a4fe`)
-
-.. _release-1.0.1:
-
-Scrapy 1.0.1 (2015-07-01)
--------------------------
-
-- Unquote request path before passing to FTPClient, it already escape paths (:commit:`cc00ad2`)
-- include tests/ to source distribution in MANIFEST.in (:commit:`eca227e`)
-- DOC Fix SelectJmes documentation (:commit:`b8567bc`)
-- DOC Bring Ubuntu and Archlinux outside of Windows subsection (:commit:`392233f`)
-- DOC remove version suffix from Ubuntu package (:commit:`5303c66`)
-- DOC Update release date for 1.0 (:commit:`c89fa29`)
-
-.. _release-1.0.0:
-
-Scrapy 1.0.0 (2015-06-19)
--------------------------
-
-You will find a lot of new features and bugfixes in this major release. Make
-sure to check our updated :ref:`overview <intro-overview>` to get a glance of
-some of the changes, along with our brushed :ref:`tutorial <intro-tutorial>`.
-
-Support for returning dictionaries in spiders
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Declaring and returning Scrapy Items is no longer necessary to collect the
-scraped data from your spider, you can now return explicit dictionaries
-instead.
-
-*Classic version*
-
-::
-
- class MyItem(scrapy.Item):
- url = scrapy.Field()
-
- class MySpider(scrapy.Spider):
- def parse(self, response):
- return MyItem(url=response.url)
-
-*New version*
-
-::
-
- class MySpider(scrapy.Spider):
- def parse(self, response):
- return {'url': response.url}
-
-Per-spider settings (GSoC 2014)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Last Google Summer of Code project accomplished an important redesign of the
-mechanism used for populating settings, introducing explicit priorities to
-override any given setting. As an extension of that goal, we included a new
-level of priority for settings that act exclusively for a single spider,
-allowing them to redefine project settings.
-
-Start using it by defining a :attr:`~scrapy.spiders.Spider.custom_settings`
-class variable in your spider::
-
- class MySpider(scrapy.Spider):
- custom_settings = {
- "DOWNLOAD_DELAY": 5.0,
- "RETRY_ENABLED": False,
- }
-
-Read more about settings population: :ref:`topics-settings`
-
-Python Logging
-~~~~~~~~~~~~~~
-
-Scrapy 1.0 has moved away from Twisted logging to support Python built in’s
-as default logging system. We’re maintaining backward compatibility for most
-of the old custom interface to call logging functions, but you’ll get
-warnings to switch to the Python logging API entirely.
-
-*Old version*
-
-::
-
- from scrapy import log
- log.msg('MESSAGE', log.INFO)
-
-*New version*
-
-::
-
- import logging
- logging.info('MESSAGE')
-
-Logging with spiders remains the same, but on top of the
-:meth:`~scrapy.spiders.Spider.log` method you’ll have access to a custom
-:attr:`~scrapy.spiders.Spider.logger` created for the spider to issue log
-events:
-
-::
-
- class MySpider(scrapy.Spider):
- def parse(self, response):
- self.logger.info('Response received')
-
-Read more in the logging documentation: :ref:`topics-logging`
-
-Crawler API refactoring (GSoC 2014)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Another milestone for last Google Summer of Code was a refactoring of the
-internal API, seeking a simpler and easier usage. Check new core interface
-in: :ref:`topics-api`
-
-A common situation where you will face these changes is while running Scrapy
-from scripts. Here’s a quick example of how to run a Spider manually with the
-new API:
-
-::
-
- from scrapy.crawler import CrawlerProcess
-
- process = CrawlerProcess({
- 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
- })
- process.crawl(MySpider)
- process.start()
-
-Bear in mind this feature is still under development and its API may change
-until it reaches a stable status.
-
-See more examples for scripts running Scrapy: :ref:`topics-practices`
-
-.. _module-relocations:
-
-Module Relocations
-~~~~~~~~~~~~~~~~~~
-
-There’s been a large rearrangement of modules trying to improve the general
-structure of Scrapy. Main changes were separating various subpackages into
-new projects and dissolving both ``scrapy.contrib`` and ``scrapy.contrib_exp``
-into top level packages. Backward compatibility was kept among internal
-relocations, while importing deprecated modules expect warnings indicating
-their new place.
-
-Full list of relocations
-************************
-
-Outsourced packages
-
-.. note::
- These extensions went through some minor changes, e.g. some setting names
- were changed. Please check the documentation in each new repository to
- get familiar with the new usage.
-
-+-------------------------------------+-------------------------------------+
-| Old location | New location |
-+=====================================+=====================================+
-| scrapy.commands.deploy | `scrapyd-client <https://github.com |
-| | /scrapy/scrapyd-client>`_ |
-| | (See other alternatives here: |
-| | :ref:`topics-deploy`) |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.djangoitem | `scrapy-djangoitem <https://github. |
-| | com/scrapy-plugins/scrapy-djangoite |
-| | m>`_ |
-+-------------------------------------+-------------------------------------+
-| scrapy.webservice | `scrapy-jsonrpc <https://github.com |
-| | /scrapy-plugins/scrapy-jsonrpc>`_ |
-+-------------------------------------+-------------------------------------+
-
-``scrapy.contrib_exp`` and ``scrapy.contrib`` dissolutions
-
-+-------------------------------------+-------------------------------------+
-| Old location | New location |
-+=====================================+=====================================+
-| scrapy.contrib\_exp.downloadermidd\ | scrapy.downloadermiddlewares.decom\ |
-| leware.decompression | pression |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib\_exp.iterators | scrapy.utils.iterators |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.downloadermiddleware | scrapy.downloadermiddlewares |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.exporter | scrapy.exporters |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.linkextractors | scrapy.linkextractors |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.loader | scrapy.loader |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.loader.processor | scrapy.loader.processors |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.pipeline | scrapy.pipelines |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.spidermiddleware | scrapy.spidermiddlewares |
-+-------------------------------------+-------------------------------------+
-| scrapy.contrib.spiders | scrapy.spiders |
-+-------------------------------------+-------------------------------------+
-| * scrapy.contrib.closespider | scrapy.extensions.\* |
-| * scrapy.contrib.corestats | |
-| * scrapy.contrib.debug | |
-| * scrapy.contrib.feedexport | |
-| * scrapy.contrib.httpcache | |
-| * scrapy.contrib.logstats | |
-| * scrapy.contrib.memdebug | |
-| * scrapy.contrib.memusage | |
-| * scrapy.contrib.spiderstate | |
-| * scrapy.contrib.statsmailer | |
-| * scrapy.contrib.throttle | |
-+-------------------------------------+-------------------------------------+
-
-Plural renames and Modules unification
-
-+-------------------------------------+-------------------------------------+
-| Old location | New location |
-+=====================================+=====================================+
-| scrapy.command | scrapy.commands |
-+-------------------------------------+-------------------------------------+
-| scrapy.dupefilter | scrapy.dupefilters |
-+-------------------------------------+-------------------------------------+
-| scrapy.linkextractor | scrapy.linkextractors |
-+-------------------------------------+-------------------------------------+
-| scrapy.spider | scrapy.spiders |
-+-------------------------------------+-------------------------------------+
-| scrapy.squeue | scrapy.squeues |
-+-------------------------------------+-------------------------------------+
-| scrapy.statscol | scrapy.statscollectors |
-+-------------------------------------+-------------------------------------+
-| scrapy.utils.decorator | scrapy.utils.decorators |
-+-------------------------------------+-------------------------------------+
-
-Class renames
-
-+-------------------------------------+-------------------------------------+
-| Old location | New location |
-+=====================================+=====================================+
-| scrapy.spidermanager.SpiderManager | scrapy.spiderloader.SpiderLoader |
-+-------------------------------------+-------------------------------------+
-
-Settings renames
-
-+-------------------------------------+-------------------------------------+
-| Old location | New location |
-+=====================================+=====================================+
-| SPIDER\_MANAGER\_CLASS | SPIDER\_LOADER\_CLASS |
-+-------------------------------------+-------------------------------------+
-
-Changelog
-~~~~~~~~~
-
-New Features and Enhancements
-
-- Python logging (:gh:`1060`, :gh:`1235`, :gh:`1236`, :gh:`1240`,
- :gh:`1259`, :gh:`1278`, :gh:`1286`)
-- FEED_EXPORT_FIELDS option (:gh:`1159`, :gh:`1224`)
-- Dns cache size and timeout options (:gh:`1132`)
-- support namespace prefix in xmliter_lxml (:gh:`963`)
-- Reactor threadpool max size setting (:gh:`1123`)
-- Allow spiders to return dicts. (:gh:`1081`)
-- Add Response.urljoin() helper (:gh:`1086`)
-- look in ~/.config/scrapy.cfg for user config (:gh:`1098`)
-- handle TLS SNI (:gh:`1101`)
-- Selectorlist extract first (:gh:`624`, :gh:`1145`)
-- Added JmesSelect (:gh:`1016`)
-- add gzip compression to filesystem http cache backend (:gh:`1020`)
-- CSS support in link extractors (:gh:`983`)
-- httpcache dont_cache meta #19 #689 (:gh:`821`)
-- add signal to be sent when request is dropped by the scheduler
- (:gh:`961`)
-- avoid download large response (:gh:`946`)
-- Allow to specify the quotechar in CSVFeedSpider (:gh:`882`)
-- Add referer to "Spider error processing" log message (:gh:`795`)
-- process robots.txt once (:gh:`896`)
-- GSoC Per-spider settings (:gh:`854`)
-- Add project name validation (:gh:`817`)
-- GSoC API cleanup (:gh:`816`, :gh:`1128`, :gh:`1147`,
- :gh:`1148`, :gh:`1156`, :gh:`1185`, :gh:`1187`, :gh:`1258`,
- :gh:`1268`, :gh:`1276`, :gh:`1285`, :gh:`1284`)
-- Be more responsive with IO operations (:gh:`1074` and :gh:`1075`)
-- Do leveldb compaction for httpcache on closing (:gh:`1297`)
-
-Deprecations and Removals
-
-- Deprecate htmlparser link extractor (:gh:`1205`)
-- remove deprecated code from FeedExporter (:gh:`1155`)
-- a leftover for.15 compatibility (:gh:`925`)
-- drop support for CONCURRENT_REQUESTS_PER_SPIDER (:gh:`895`)
-- Drop old engine code (:gh:`911`)
-- Deprecate SgmlLinkExtractor (:gh:`777`)
-
-Relocations
-
-- Move exporters/__init__.py to exporters.py (:gh:`1242`)
-- Move base classes to their packages (:gh:`1218`, :gh:`1233`)
-- Module relocation (:gh:`1181`, :gh:`1210`)
-- rename SpiderManager to SpiderLoader (:gh:`1166`)
-- Remove djangoitem (:gh:`1177`)
-- remove scrapy deploy command (:gh:`1102`)
-- dissolve contrib_exp (:gh:`1134`)
-- Deleted bin folder from root, fixes #913 (:gh:`914`)
-- Remove jsonrpc based webservice (:gh:`859`)
-- Move Test cases under project root dir (:gh:`827`, :gh:`841`)
-- Fix backward incompatibility for relocated paths in settings
- (:gh:`1267`)
-
-Documentation
-
-- CrawlerProcess documentation (:gh:`1190`)
-- Favoring web scraping over screen scraping in the descriptions
- (:gh:`1188`)
-- Some improvements for Scrapy tutorial (:gh:`1180`)
-- Documenting Files Pipeline together with Images Pipeline (:gh:`1150`)
-- deployment docs tweaks (:gh:`1164`)
-- Added deployment section covering scrapyd-deploy and shub (:gh:`1124`)
-- Adding more settings to project template (:gh:`1073`)
-- some improvements to overview page (:gh:`1106`)
-- Updated link in docs/topics/architecture.rst (:gh:`647`)
-- DOC reorder topics (:gh:`1022`)
-- updating list of Request.meta special keys (:gh:`1071`)
-- DOC document download_timeout (:gh:`898`)
-- DOC simplify extension docs (:gh:`893`)
-- Leaks docs (:gh:`894`)
-- DOC document from_crawler method for item pipelines (:gh:`904`)
-- Spider_error doesn't support deferreds (:gh:`1292`)
-- Corrections & Sphinx related fixes (:gh:`1220`, :gh:`1219`,
- :gh:`1196`, :gh:`1172`, :gh:`1171`, :gh:`1169`, :gh:`1160`,
- :gh:`1154`, :gh:`1127`, :gh:`1112`, :gh:`1105`, :gh:`1041`,
- :gh:`1082`, :gh:`1033`, :gh:`944`, :gh:`866`, :gh:`864`,
- :gh:`796`, :gh:`1260`, :gh:`1271`, :gh:`1293`, :gh:`1298`)
-
-Bugfixes
-
-- Item multi inheritance fix (:gh:`353`, :gh:`1228`)
-- ItemLoader.load_item: iterate over copy of fields (:gh:`722`)
-- Fix Unhandled error in Deferred (RobotsTxtMiddleware) (:gh:`1131`,
- :gh:`1197`)
-- Force to read DOWNLOAD_TIMEOUT as int (:gh:`954`)
-- scrapy.utils.misc.load_object should print full traceback (:gh:`902`)
-- Fix bug for ".local" host name (:gh:`878`)
-- Fix for Enabled extensions, middlewares, pipelines info not printed
- anymore (:gh:`879`)
-- fix dont_merge_cookies bad behaviour when set to false on meta
- (:gh:`846`)
-
-Python 3 In Progress Support
-
-- disable scrapy.telnet if twisted.conch is not available (:gh:`1161`)
-- fix Python 3 syntax errors in ajaxcrawl.py (:gh:`1162`)
-- more python3 compatibility changes for urllib (:gh:`1121`)
-- assertItemsEqual was renamed to assertCountEqual in Python 3.
- (:gh:`1070`)
-- Import unittest.mock if available. (:gh:`1066`)
-- updated deprecated cgi.parse_qsl to use six's parse_qsl (:gh:`909`)
-- Prevent Python 3 port regressions (:gh:`830`)
-- PY3: use MutableMapping for python 3 (:gh:`810`)
-- PY3: use six.BytesIO and six.moves.cStringIO (:gh:`803`)
-- PY3: fix xmlrpclib and email imports (:gh:`801`)
-- PY3: use six for robotparser and urlparse (:gh:`800`)
-- PY3: use six.iterkeys, six.iteritems, and tempfile (:gh:`799`)
-- PY3: fix has_key and use six.moves.configparser (:gh:`798`)
-- PY3: use six.moves.cPickle (:gh:`797`)
-- PY3 make it possible to run some tests in Python3 (:gh:`776`)
-
-Tests
-
-- remove unnecessary lines from py3-ignores (:gh:`1243`)
-- Fix remaining warnings from pytest while collecting tests (:gh:`1206`)
-- Add docs build to travis (:gh:`1234`)
-- TST don't collect tests from deprecated modules. (:gh:`1165`)
-- install service_identity package in tests to prevent warnings
- (:gh:`1168`)
-- Fix deprecated settings API in tests (:gh:`1152`)
-- Add test for webclient with POST method and no body given (:gh:`1089`)
-- py3-ignores.txt supports comments (:gh:`1044`)
-- modernize some of the asserts (:gh:`835`)
-- selector.__repr__ test (:gh:`779`)
-
-Code refactoring
-
-- CSVFeedSpider cleanup: use iterate_spider_output (:gh:`1079`)
-- remove unnecessary check from scrapy.utils.spider.iter_spider_output
- (:gh:`1078`)
-- Pydispatch pep8 (:gh:`992`)
-- Removed unused 'load=False' parameter from walk_modules() (:gh:`871`)
-- For consistency, use ``job_dir`` helper in ``SpiderState`` extension.
- (:gh:`805`)
-- rename "sflo" local variables to less cryptic "log_observer" (:gh:`775`)
-
-Scrapy 0.24.6 (2015-04-20)
---------------------------
-
-- encode invalid xpath with unicode_escape under PY2 (:commit:`07cb3e5`)
-- fix IPython shell scope issue and load IPython user config (:commit:`2c8e573`)
-- Fix small typo in the docs (:commit:`d694019`)
-- Fix small typo (:commit:`f92fa83`)
-- Converted sel.xpath() calls to response.xpath() in Extracting the data (:commit:`c2c6d15`)
-
-
-Scrapy 0.24.5 (2015-02-25)
---------------------------
-
-- Support new _getEndpoint Agent signatures on Twisted 15.0.0 (:commit:`540b9bc`)
-- DOC a couple more references are fixed (:commit:`b4c454b`)
-- DOC fix a reference (:commit:`e3c1260`)
-- t.i.b.ThreadedResolver is now a new-style class (:commit:`9e13f42`)
-- S3DownloadHandler: fix auth for requests with quoted paths/query params (:commit:`cdb9a0b`)
-- fixed the variable types in mailsender documentation (:commit:`bb3a848`)
-- Reset items_scraped instead of item_count (:commit:`edb07a4`)
-- Tentative attention message about what document to read for contributions (:commit:`7ee6f7a`)
-- mitmproxy 0.10.1 needs netlib 0.10.1 too (:commit:`874fcdd`)
-- pin mitmproxy 0.10.1 as >0.11 does not work with tests (:commit:`c6b21f0`)
-- Test the parse command locally instead of against an external url (:commit:`c3a6628`)
-- Patches Twisted issue while closing the connection pool on HTTPDownloadHandler (:commit:`d0bf957`)
-- Updates documentation on dynamic item classes. (:commit:`eeb589a`)
-- Merge pull request #943 from Lazar-T/patch-3 (:commit:`5fdab02`)
-- typo (:commit:`b0ae199`)
-- pywin32 is required by Twisted. closes #937 (:commit:`5cb0cfb`)
-- Update install.rst (:commit:`781286b`)
-- Merge pull request #928 from Lazar-T/patch-1 (:commit:`b415d04`)
-- comma instead of fullstop (:commit:`627b9ba`)
-- Merge pull request #885 from jsma/patch-1 (:commit:`de909ad`)
-- Update request-response.rst (:commit:`3f3263d`)
-- SgmlLinkExtractor - fix for parsing <area> tag with Unicode present (:commit:`49b40f0`)
-
-Scrapy 0.24.4 (2014-08-09)
---------------------------
-
-- pem file is used by mockserver and required by scrapy bench (:commit:`5eddc68b63`)
-- scrapy bench needs scrapy.tests* (:commit:`d6cb999`)
-
-Scrapy 0.24.3 (2014-08-09)
---------------------------
-
-- no need to waste travis-ci time on py3 for 0.24 (:commit:`8e080c1`)
-- Update installation docs (:commit:`1d0c096`)
-- There is a trove classifier for Scrapy framework! (:commit:`4c701d7`)
-- update other places where w3lib version is mentioned (:commit:`d109c13`)
-- Update w3lib requirement to 1.8.0 (:commit:`39d2ce5`)
-- Use w3lib.html.replace_entities() (remove_entities() is deprecated) (:commit:`180d3ad`)
-- set zip_safe=False (:commit:`a51ee8b`)
-- do not ship tests package (:commit:`ee3b371`)
-- scrapy.bat is not needed anymore (:commit:`c3861cf`)
-- Modernize setup.py (:commit:`362e322`)
-- headers can not handle non-string values (:commit:`94a5c65`)
-- fix ftp test cases (:commit:`a274a7f`)
-- The sum up of travis-ci builds are taking like 50min to complete (:commit:`ae1e2cc`)
-- Update shell.rst typo (:commit:`e49c96a`)
-- removes weird indentation in the shell results (:commit:`1ca489d`)
-- improved explanations, clarified blog post as source, added link for XPath string functions in the spec (:commit:`65c8f05`)
-- renamed UserTimeoutError and ServerTimeouterror #583 (:commit:`037f6ab`)
-- adding some xpath tips to selectors docs (:commit:`2d103e0`)
-- fix tests to account for https://github.com/scrapy/w3lib/pull/23 (:commit:`f8d366a`)
-- get_func_args maximum recursion fix #728 (:commit:`81344ea`)
-- Updated input/output processor example according to #560. (:commit:`f7c4ea8`)
-- Fixed Python syntax in tutorial. (:commit:`db59ed9`)
-- Add test case for tunneling proxy (:commit:`f090260`)
-- Bugfix for leaking Proxy-Authorization header to remote host when using tunneling (:commit:`d8793af`)
-- Extract links from XHTML documents with MIME-Type "application/xml" (:commit:`ed1f376`)
-- Merge pull request #793 from roysc/patch-1 (:commit:`91a1106`)
-- Fix typo in commands.rst (:commit:`743e1e2`)
-- better testcase for settings.overrides.setdefault (:commit:`e22daaf`)
-- Using CRLF as line marker according to http 1.1 definition (:commit:`5ec430b`)
-
-Scrapy 0.24.2 (2014-07-08)
---------------------------
-
-- Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute (:commit:`e5e8133`)
-- there is not support for python3 yet (:commit:`3cd6146`)
-- Update python compatible version set to Debian packages (:commit:`fa5d76b`)
-- DOC fix formatting in release notes (:commit:`c6a9e20`)
-
-Scrapy 0.24.1 (2014-06-27)
---------------------------
-
-- Fix deprecated CrawlerSettings and increase backward compatibility with
- .defaults attribute (:commit:`8e3f20a`)
-
-
-Scrapy 0.24.0 (2014-06-26)
---------------------------
-
-Enhancements
-~~~~~~~~~~~~
-
-- Improve Scrapy top-level namespace (:gh:`494`, :gh:`684`)
-- Add selector shortcuts to responses (:gh:`554`, :gh:`690`)
-- Add new lxml based LinkExtractor to replace unmaintained SgmlLinkExtractor
- (:gh:`559`, :gh:`761`, :gh:`763`)
-- Cleanup settings API - part of per-spider settings **GSoC project** (:gh:`737`)
-- Add UTF8 encoding header to templates (:gh:`688`, :gh:`762`)
-- Telnet console now binds to 127.0.0.1 by default (:gh:`699`)
-- Update Debian/Ubuntu install instructions (:gh:`509`, :gh:`549`)
-- Disable smart strings in lxml XPath evaluations (:gh:`535`)
-- Restore filesystem based cache as default for http
- cache middleware (:gh:`541`, :gh:`500`, :gh:`571`)
-- Expose current crawler in Scrapy shell (:gh:`557`)
-- Improve testsuite comparing CSV and XML exporters (:gh:`570`)
-- New ``offsite/filtered`` and ``offsite/domains`` stats (:gh:`566`)
-- Support process_links as generator in CrawlSpider (:gh:`555`)
-- Verbose logging and new stats counters for DupeFilter (:gh:`553`)
-- Add a mimetype parameter to ``MailSender.send()`` (:gh:`602`)
-- Generalize file pipeline log messages (:gh:`622`)
-- Replace unencodeable codepoints with html entities in SGMLLinkExtractor (:gh:`565`)
-- Converted SEP documents to rst format (:gh:`629`, :gh:`630`,
- :gh:`638`, :gh:`632`, :gh:`636`, :gh:`640`, :gh:`635`,
- :gh:`634`, :gh:`639`, :gh:`637`, :gh:`631`, :gh:`633`,
- :gh:`641`, :gh:`642`)
-- Tests and docs for clickdata's nr index in FormRequest (:gh:`646`, :gh:`645`)
-- Allow to disable a downloader handler just like any other component (:gh:`650`)
-- Log when a request is discarded after too many redirections (:gh:`654`)
-- Log error responses if they are not handled by spider callbacks
- (:gh:`612`, :gh:`656`)
-- Add content-type check to http compression mw (:gh:`193`, :gh:`660`)
-- Run pypy tests using latest pypi from ppa (:gh:`674`)
-- Run test suite using pytest instead of trial (:gh:`679`)
-- Build docs and check for dead links in tox environment (:gh:`687`)
-- Make scrapy.version_info a tuple of integers (:gh:`681`, :gh:`692`)
-- Infer exporter's output format from filename extensions
- (:gh:`546`, :gh:`659`, :gh:`760`)
-- Support case-insensitive domains in ``url_is_from_any_domain()`` (:gh:`693`)
-- Remove pep8 warnings in project and spider templates (:gh:`698`)
-- Tests and docs for ``request_fingerprint`` function (:gh:`597`)
-- Update SEP-19 for GSoC project ``per-spider settings`` (:gh:`705`)
-- Set exit code to non-zero when contracts fails (:gh:`727`)
-- Add a setting to control what class is instantiated as Downloader component
- (:gh:`738`)
-- Pass response in ``item_dropped`` signal (:gh:`724`)
-- Improve ``scrapy check`` contracts command (:gh:`733`, :gh:`752`)
-- Document ``spider.closed()`` shortcut (:gh:`719`)
-- Document ``request_scheduled`` signal (:gh:`746`)
-- Add a note about reporting security issues (:gh:`697`)
-- Add LevelDB http cache storage backend (:gh:`626`, :gh:`500`)
-- Sort spider list output of ``scrapy list`` command (:gh:`742`)
-- Multiple documentation enhancements and fixes
- (:gh:`575`, :gh:`587`, :gh:`590`, :gh:`596`, :gh:`610`,
- :gh:`617`, :gh:`618`, :gh:`627`, :gh:`613`, :gh:`643`,
- :gh:`654`, :gh:`675`, :gh:`663`, :gh:`711`, :gh:`714`)
-
-Bugfixes
-~~~~~~~~
-
-- Encode unicode URL value when creating Links in RegexLinkExtractor (:gh:`561`)
-- Ignore None values in ItemLoader processors (:gh:`556`)
-- Fix link text when there is an inner tag in SGMLLinkExtractor and
- HtmlParserLinkExtractor (:gh:`485`, :gh:`574`)
-- Fix wrong checks on subclassing of deprecated classes
- (:gh:`581`, :gh:`584`)
-- Handle errors caused by inspect.stack() failures (:gh:`582`)
-- Fix a reference to unexistent engine attribute (:gh:`593`, :gh:`594`)
-- Fix dynamic itemclass example usage of type() (:gh:`603`)
-- Use lucasdemarchi/codespell to fix typos (:gh:`628`)
-- Fix default value of attrs argument in SgmlLinkExtractor to be tuple (:gh:`661`)
-- Fix XXE flaw in sitemap reader (:gh:`676`)
-- Fix engine to support filtered start requests (:gh:`707`)
-- Fix offsite middleware case on urls with no hostnames (:gh:`745`)
-- Testsuite doesn't require PIL anymore (:gh:`585`)
-
-
-Scrapy 0.22.2 (released 2014-02-14)
------------------------------------
-
-- fix a reference to unexistent engine.slots. closes #593 (:commit:`13c099a`)
-- downloaderMW doc typo (spiderMW doc copy remnant) (:commit:`8ae11bf`)
-- Correct typos (:commit:`1346037`)
-
-Scrapy 0.22.1 (released 2014-02-08)
------------------------------------
-
-- localhost666 can resolve under certain circumstances (:commit:`2ec2279`)
-- test inspect.stack failure (:commit:`cc3eda3`)
-- Handle cases when inspect.stack() fails (:commit:`8cb44f9`)
-- Fix wrong checks on subclassing of deprecated classes. closes #581 (:commit:`46d98d6`)
-- Docs: 4-space indent for final spider example (:commit:`13846de`)
-- Fix HtmlParserLinkExtractor and tests after #485 merge (:commit:`368a946`)
-- BaseSgmlLinkExtractor: Fixed the missing space when the link has an inner tag (:commit:`b566388`)
-- BaseSgmlLinkExtractor: Added unit test of a link with an inner tag (:commit:`c1cb418`)
-- BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag (:commit:`7e4d627`)
-- Fix tests for Travis-CI build (:commit:`76c7e20`)
-- replace unencodeable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)
-- RegexLinkExtractor: encode URL unicode value when creating Links (:commit:`d0ee545`)
-- Updated the tutorial crawl output with latest output. (:commit:`8da65de`)
-- Updated shell docs with the crawler reference and fixed the actual shell output. (:commit:`875b9ab`)
-- PEP8 minor edits. (:commit:`f89efaf`)
-- Expose current crawler in the Scrapy shell. (:commit:`5349cec`)
-- Unused re import and PEP8 minor edits. (:commit:`387f414`)
-- Ignore None's values when using the ItemLoader. (:commit:`0632546`)
-- DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. (:commit:`cde9a8c`)
-- show Ubuntu setup instructions as literal code (:commit:`fb5c9c5`)
-- Update Ubuntu installation instructions (:commit:`70fb105`)
-- Merge pull request #550 from stray-leone/patch-1 (:commit:`6f70b6a`)
-- modify the version of Scrapy Ubuntu package (:commit:`725900d`)
-- fix 0.22.0 release date (:commit:`af0219a`)
-- fix typos in news.rst and remove (not released yet) header (:commit:`b7f58f4`)
-
-Scrapy 0.22.0 (released 2014-01-17)
------------------------------------
-
-Enhancements
-~~~~~~~~~~~~
-
-- [**Backward incompatible**] Switched HTTPCacheMiddleware backend to filesystem (:gh:`541`)
- To restore old backend set ``HTTPCACHE_STORAGE`` to ``scrapy.contrib.httpcache.DbmCacheStorage``
-- Proxy \https:// urls using CONNECT method (:gh:`392`, :gh:`397`)
-- Add a middleware to crawl ajax crawlable pages as defined by google (:gh:`343`)
-- Rename scrapy.spider.BaseSpider to scrapy.spider.Spider (:gh:`510`, :gh:`519`)
-- Selectors register EXSLT namespaces by default (:gh:`472`)
-- Unify item loaders similar to selectors renaming (:gh:`461`)
-- Make ``RFPDupeFilter`` class easily subclassable (:gh:`533`)
-- Improve test coverage and forthcoming Python 3 support (:gh:`525`)
-- Promote startup info on settings and middleware to INFO level (:gh:`520`)
-- Support partials in ``get_func_args`` util (:gh:`506`, :gh:`504`)
-- Allow running individual tests via tox (:gh:`503`)
-- Update extensions ignored by link extractors (:gh:`498`)
-- Add middleware methods to get files/images/thumbs paths (:gh:`490`)
-- Improve offsite middleware tests (:gh:`478`)
-- Add a way to skip default Referer header set by RefererMiddleware (:gh:`475`)
-- Do not send ``x-gzip`` in default ``Accept-Encoding`` header (:gh:`469`)
-- Support defining http error handling using settings (:gh:`466`)
-- Use modern python idioms wherever you find legacies (:gh:`497`)
-- Improve and correct documentation
- (:gh:`527`, :gh:`524`, :gh:`521`, :gh:`517`, :gh:`512`, :gh:`505`,
- :gh:`502`, :gh:`489`, :gh:`465`, :gh:`460`, :gh:`425`, :gh:`536`)
-
-Fixes
-~~~~~
-
-- Update Selector class imports in CrawlSpider template (:gh:`484`)
-- Fix unexistent reference to ``engine.slots`` (:gh:`464`)
-- Do not try to call ``body_as_unicode()`` on a non-TextResponse instance (:gh:`462`)
-- Warn when subclassing XPathItemLoader, previously it only warned on
- instantiation. (:gh:`523`)
-- Warn when subclassing XPathSelector, previously it only warned on
- instantiation. (:gh:`537`)
-- Multiple fixes to memory stats (:gh:`531`, :gh:`530`, :gh:`529`)
-- Fix overriding url in ``FormRequest.from_response()`` (:gh:`507`)
-- Fix tests runner under pip 1.5 (:gh:`513`)
-- Fix logging error when spider name is unicode (:gh:`479`)
-
-Scrapy 0.20.2 (released 2013-12-09)
------------------------------------
-
-- Update CrawlSpider Template with Selector changes (:commit:`6d1457d`)
-- fix method name in tutorial. closes GH-480 (:commit:`b4fc359`
-
-Scrapy 0.20.1 (released 2013-11-28)
------------------------------------
-
-- include_package_data is required to build wheels from published sources (:commit:`5ba1ad5`)
-- process_parallel was leaking the failures on its internal deferreds. closes #458 (:commit:`419a780`)
-
-Scrapy 0.20.0 (released 2013-11-08)
------------------------------------
-
-Enhancements
-~~~~~~~~~~~~
-
-- New Selector's API including CSS selectors (:gh:`395` and :gh:`426`),
-- Request/Response url/body attributes are now immutable
- (modifying them had been deprecated for a long time)
-- :setting:`ITEM_PIPELINES` is now defined as a dict (instead of a list)
-- Sitemap spider can fetch alternate URLs (:gh:`360`)
-- ``Selector.remove_namespaces()`` now remove namespaces from element's attributes. (:gh:`416`)
-- Paved the road for Python 3.3+ (:gh:`435`, :gh:`436`, :gh:`431`, :gh:`452`)
-- New item exporter using native python types with nesting support (:gh:`366`)
-- Tune HTTP1.1 pool size so it matches concurrency defined by settings (:commit:`b43b5f575`)
-- scrapy.mail.MailSender now can connect over TLS or upgrade using STARTTLS (:gh:`327`)
-- New FilesPipeline with functionality factored out from ImagesPipeline (:gh:`370`, :gh:`409`)
-- Recommend Pillow instead of PIL for image handling (:gh:`317`)
-- Added Debian packages for Ubuntu Quantal and Raring (:commit:`86230c0`)
-- Mock server (used for tests) can listen for HTTPS requests (:gh:`410`)
-- Remove multi spider support from multiple core components
- (:gh:`422`, :gh:`421`, :gh:`420`, :gh:`419`, :gh:`423`, :gh:`418`)
-- Travis-CI now tests Scrapy changes against development versions of ``w3lib`` and ``queuelib`` python packages.
-- Add pypy 2.1 to continuous integration tests (:commit:`ecfa7431`)
-- Pylinted, pep8 and removed old-style exceptions from source (:gh:`430`, :gh:`432`)
-- Use importlib for parametric imports (:gh:`445`)
-- Handle a regression introduced in Python 2.7.5 that affects XmlItemExporter (:gh:`372`)
-- Bugfix crawling shutdown on SIGINT (:gh:`450`)
-- Do not submit ``reset`` type inputs in FormRequest.from_response (:commit:`b326b87`)
-- Do not silence download errors when request errback raises an exception (:commit:`684cfc0`)
-
-Bugfixes
-~~~~~~~~
-
-- Fix tests under Django 1.6 (:commit:`b6bed44c`)
-- Lot of bugfixes to retry middleware under disconnections using HTTP 1.1 download handler
-- Fix inconsistencies among Twisted releases (:gh:`406`)
-- Fix Scrapy shell bugs (:gh:`418`, :gh:`407`)
-- Fix invalid variable name in setup.py (:gh:`429`)
-- Fix tutorial references (:gh:`387`)
-- Improve request-response docs (:gh:`391`)
-- Improve best practices docs (:gh:`399`, :gh:`400`, :gh:`401`, :gh:`402`)
-- Improve django integration docs (:gh:`404`)
-- Document ``bindaddress`` request meta (:commit:`37c24e01d7`)
-- Improve ``Request`` class documentation (:gh:`226`)
-
-Other
-~~~~~
-
-- Dropped Python 2.6 support (:gh:`448`)
-- Add :doc:`cssselect <cssselect:index>` python package as install dependency
-- Drop libxml2 and multi selector's backend support, `lxml`_ is required from now on.
-- Minimum Twisted version increased to 10.0.0, dropped Twisted 8.0 support.
-- Running test suite now requires ``mock`` python library (:gh:`390`)
-
-
-Thanks
-~~~~~~
-
-Thanks to everyone who contribute to this release!
-
-List of contributors sorted by number of commits::
-
- 69 Daniel Graña <dangra@...>
- 37 Pablo Hoffman <pablo@...>
- 13 Mikhail Korobov <kmike84@...>
- 9 Alex Cepoi <alex.cepoi@...>
- 9 alexanderlukanin13 <alexander.lukanin.13@...>
- 8 Rolando Espinoza La fuente <darkrho@...>
- 8 Lukasz Biedrycki <lukasz.biedrycki@...>
- 6 Nicolas Ramirez <nramirez.uy@...>
- 3 Paul Tremberth <paul.tremberth@...>
- 2 Martin Olveyra <molveyra@...>
- 2 Stefan <misc@...>
- 2 Rolando Espinoza <darkrho@...>
- 2 Loren Davie <loren@...>
- 2 irgmedeiros <irgmedeiros@...>
- 1 Stefan Koch <taikano@...>
- 1 Stefan <cct@...>
- 1 scraperdragon <dragon@...>
- 1 Kumara Tharmalingam <ktharmal@...>
- 1 Francesco Piccinno <stack.box@...>
- 1 Marcos Campal <duendex@...>
- 1 Dragon Dave <dragon@...>
- 1 Capi Etheriel <barraponto@...>
- 1 cacovsky <amarquesferraz@...>
- 1 Berend Iwema <berend@...>
-
-Scrapy 0.18.4 (released 2013-10-10)
------------------------------------
-
-- IPython refuses to update the namespace. fix #396 (:commit:`3d32c4f`)
-- Fix AlreadyCalledError replacing a request in shell command. closes #407 (:commit:`b1d8919`)
-- Fix ``start_requests()`` laziness and early hangs (:commit:`89faf52`)
-
-Scrapy 0.18.3 (released 2013-10-03)
------------------------------------
-
-- fix regression on lazy evaluation of start requests (:commit:`12693a5`)
-- forms: do not submit reset inputs (:commit:`e429f63`)
-- increase unittest timeouts to decrease travis false positive failures (:commit:`912202e`)
-- backport master fixes to json exporter (:commit:`cfc2d46`)
-- Fix permission and set umask before generating sdist tarball (:commit:`06149e0`)
-
-Scrapy 0.18.2 (released 2013-09-03)
------------------------------------
-
-- Backport ``scrapy check`` command fixes and backward compatible multi
- crawler process(:gh:`339`)
-
-Scrapy 0.18.1 (released 2013-08-27)
------------------------------------
-
-- remove extra import added by cherry picked changes (:commit:`d20304e`)
-- fix crawling tests under twisted pre 11.0.0 (:commit:`1994f38`)
-- py26 can not format zero length fields {} (:commit:`abf756f`)
-- test PotentiaDataLoss errors on unbound responses (:commit:`b15470d`)
-- Treat responses without content-length or Transfer-Encoding as good responses (:commit:`c4bf324`)
-- do no include ResponseFailed if http11 handler is not enabled (:commit:`6cbe684`)
-- New HTTP client wraps connection lost in ResponseFailed exception. fix #373 (:commit:`1a20bba`)
-- limit travis-ci build matrix (:commit:`3b01bb8`)
-- Merge pull request #375 from peterarenot/patch-1 (:commit:`fa766d7`)
-- Fixed so it refers to the correct folder (:commit:`3283809`)
-- added Quantal & Raring to support Ubuntu releases (:commit:`1411923`)
-- fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 (:commit:`bb35ed0`)
-- fix XmlItemExporter in Python 2.7.4 and 2.7.5 (:commit:`de3e451`)
-- minor updates to 0.18 release notes (:commit:`c45e5f1`)
-- fix contributors list format (:commit:`0b60031`)
-
-Scrapy 0.18.0 (released 2013-08-09)
------------------------------------
-
-- Lot of improvements to testsuite run using Tox, including a way to test on pypi
-- Handle GET parameters for AJAX crawlable urls (:commit:`3fe2a32`)
-- Use lxml recover option to parse sitemaps (:gh:`347`)
-- Bugfix cookie merging by hostname and not by netloc (:gh:`352`)
-- Support disabling ``HttpCompressionMiddleware`` using a flag setting (:gh:`359`)
-- Support xml namespaces using ``iternodes`` parser in ``XMLFeedSpider`` (:gh:`12`)
-- Support ``dont_cache`` request meta flag (:gh:`19`)
-- Bugfix ``scrapy.utils.gz.gunzip`` broken by changes in python 2.7.4 (:commit:`4dc76e`)
-- Bugfix url encoding on ``SgmlLinkExtractor`` (:gh:`24`)
-- Bugfix ``TakeFirst`` processor shouldn't discard zero (0) value (:gh:`59`)
-- Support nested items in xml exporter (:gh:`66`)
-- Improve cookies handling performance (:gh:`77`)
-- Log dupe filtered requests once (:gh:`105`)
-- Split redirection middleware into status and meta based middlewares (:gh:`78`)
-- Use HTTP1.1 as default downloader handler (:gh:`109` and :gh:`318`)
-- Support xpath form selection on ``FormRequest.from_response`` (:gh:`185`)
-- Bugfix unicode decoding error on ``SgmlLinkExtractor`` (:gh:`199`)
-- Bugfix signal dispatching on pypi interpreter (:gh:`205`)
-- Improve request delay and concurrency handling (:gh:`206`)
-- Add RFC2616 cache policy to ``HttpCacheMiddleware`` (:gh:`212`)
-- Allow customization of messages logged by engine (:gh:`214`)
-- Multiples improvements to ``DjangoItem`` (:gh:`217`, :gh:`218`, :gh:`221`)
-- Extend Scrapy commands using setuptools entry points (:gh:`260`)
-- Allow spider ``allowed_domains`` value to be set/tuple (:gh:`261`)
-- Support ``settings.getdict`` (:gh:`269`)
-- Simplify internal ``scrapy.core.scraper`` slot handling (:gh:`271`)
-- Added ``Item.copy`` (:gh:`290`)
-- Collect idle downloader slots (:gh:`297`)
-- Add ``ftp://`` scheme downloader handler (:gh:`329`)
-- Added downloader benchmark webserver and spider tools :ref:`benchmarking`
-- Moved persistent (on disk) queues to a separate project (queuelib_) which Scrapy now depends on
-- Add Scrapy commands using external libraries (:gh:`260`)
-- Added ``--pdb`` option to ``scrapy`` command line tool
-- Added :meth:`XPathSelector.remove_namespaces <scrapy.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
-- Several improvements to spider contracts
-- New default middleware named MetaRefreshMiddleware that handles meta-refresh html tag redirections,
-- MetaRefreshMiddleware and RedirectMiddleware have different priorities to address #62
-- added from_crawler method to spiders
-- added system tests with mock server
-- more improvements to macOS compatibility (thanks Alex Cepoi)
-- several more cleanups to singletons and multi-spider support (thanks Nicolas Ramirez)
-- support custom download slots
-- added --spider option to "shell" command.
-- log overridden settings when Scrapy starts
-
-Thanks to everyone who contribute to this release. Here is a list of
-contributors sorted by number of commits::
-
- 130 Pablo Hoffman <pablo@...>
- 97 Daniel Graña <dangra@...>
- 20 Nicolás Ramírez <nramirez.uy@...>
- 13 Mikhail Korobov <kmike84@...>
- 12 Pedro Faustino <pedrobandim@...>
- 11 Steven Almeroth <sroth77@...>
- 5 Rolando Espinoza La fuente <darkrho@...>
- 4 Michal Danilak <mimino.coder@...>
- 4 Alex Cepoi <alex.cepoi@...>
- 4 Alexandr N Zamaraev (aka tonal) <tonal@...>
- 3 paul <paul.tremberth@...>
- 3 Martin Olveyra <molveyra@...>
- 3 Jordi Llonch <llonchj@...>
- 3 arijitchakraborty <myself.arijit@...>
- 2 Shane Evans <shane.evans@...>
- 2 joehillen <joehillen@...>
- 2 Hart <HartSimha@...>
- 2 Dan <ellisd23@...>
- 1 Zuhao Wan <wanzuhao@...>
- 1 whodatninja <blake@...>
- 1 vkrest <v.krestiannykov@...>
- 1 tpeng <pengtaoo@...>
- 1 Tom Mortimer-Jones <tom@...>
- 1 Rocio Aramberri <roschegel@...>
- 1 Pedro <pedro@...>
- 1 notsobad <wangxiaohugg@...>
- 1 Natan L <kuyanatan.nlao@...>
- 1 Mark Grey <mark.grey@...>
- 1 Luan <luanpab@...>
- 1 Libor Nenadál <libor.nenadal@...>
- 1 Juan M Uys <opyate@...>
- 1 Jonas Brunsgaard <jonas.brunsgaard@...>
- 1 Ilya Baryshev <baryshev@...>
- 1 Hasnain Lakhani <m.hasnain.lakhani@...>
- 1 Emanuel Schorsch <emschorsch@...>
- 1 Chris Tilden <chris.tilden@...>
- 1 Capi Etheriel <barraponto@...>
- 1 cacovsky <amarquesferraz@...>
- 1 Berend Iwema <berend@...>
-
-
-Scrapy 0.16.5 (released 2013-05-30)
------------------------------------
-
-- obey request method when Scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)
-- fix inaccurate downloader middleware documentation. refs #280 (:commit:`40667cb`)
-- doc: remove links to diveintopython.org, which is no longer available. closes #246 (:commit:`bd58bfa`)
-- Find form nodes in invalid html5 documents (:commit:`e3d6945`)
-- Fix typo labeling attrs type bool instead of list (:commit:`a274276`)
-
-Scrapy 0.16.4 (released 2013-01-23)
------------------------------------
-
-- fixes spelling errors in documentation (:commit:`6d2b3aa`)
-- add doc about disabling an extension. refs #132 (:commit:`c90de33`)
-- Fixed error message formatting. log.err() doesn't support cool formatting and when error occurred, the message was: "ERROR: Error processing %(item)s" (:commit:`c16150c`)
-- lint and improve images pipeline error logging (:commit:`56b45fc`)
-- fixed doc typos (:commit:`243be84`)
-- add documentation topics: Broad Crawls & Common Practices (:commit:`1fbb715`)
-- fix bug in Scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)
-- Update docs/topics/commands.rst (:commit:`28eac7a`)
-
-Scrapy 0.16.3 (released 2012-12-07)
------------------------------------
-
-- Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced (:commit:`487b9b5`)
-- add error details when image pipeline fails (:commit:`8232569`)
-- improve macOS compatibility (:commit:`8dcf8aa`)
-- setup.py: use README.rst to populate long_description (:commit:`7b5310d`)
-- doc: removed obsolete references to ClientForm (:commit:`80f9bb6`)
-- correct docs for default storage backend (:commit:`2aa491b`)
-- doc: removed broken proxyhub link from FAQ (:commit:`bdf61c4`)
-- Fixed docs typo in SpiderOpenCloseLogging example (:commit:`7184094`)
-
-
-Scrapy 0.16.2 (released 2012-11-09)
------------------------------------
-
-- Scrapy contracts: python2.6 compat (:commit:`a4a9199`)
-- Scrapy contracts verbose option (:commit:`ec41673`)
-- proper unittest-like output for Scrapy contracts (:commit:`86635e4`)
-- added open_in_browser to debugging doc (:commit:`c9b690d`)
-- removed reference to global Scrapy stats from settings doc (:commit:`dd55067`)
-- Fix SpiderState bug in Windows platforms (:commit:`58998f4`)
-
-
-Scrapy 0.16.1 (released 2012-10-26)
------------------------------------
-
-- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release (:commit:`8c780fd`)
-- better backward compatibility for scrapy.conf.settings (:commit:`3403089`)
-- extended documentation on how to access crawler stats from extensions (:commit:`c4da0b5`)
-- removed .hgtags (no longer needed now that Scrapy uses git) (:commit:`d52c188`)
-- fix dashes under rst headers (:commit:`fa4f7f9`)
-- set release date for 0.16.0 in news (:commit:`e292246`)
-
-
-Scrapy 0.16.0 (released 2012-10-18)
------------------------------------
-
-Scrapy changes:
-
-- added :ref:`topics-contracts`, a mechanism for testing spiders in a formal/reproducible way
-- added options ``-o`` and ``-t`` to the :command:`runspider` command
-- documented :doc:`topics/autothrottle` and added to extensions installed by default. You still need to enable it with :setting:`AUTOTHROTTLE_ENABLED`
-- major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (``stats_spider_opened``, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.
-- added a ``process_start_requests()`` method to spider middlewares
-- dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.
-- dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.
-- documented :ref:`topics-api`
-- ``lxml`` is now the default selectors backend instead of ``libxml2``
-- ported FormRequest.from_response() to use `lxml`_ instead of `ClientForm`_
-- removed modules: ``scrapy.xlib.BeautifulSoup`` and ``scrapy.xlib.ClientForm``
-- SitemapSpider: added support for sitemap urls ending in .xml and .xml.gz, even if they advertise a wrong content type (:commit:`10ed28b`)
-- StackTraceDump extension: also dump trackref live references (:commit:`fe2ce93`)
-- nested items now fully supported in JSON and JSONLines exporters
-- added :reqmeta:`cookiejar` Request meta key to support multiple cookie sessions per spider
-- decoupled encoding detection code to `w3lib.encoding`_, and ported Scrapy code to use that module
-- dropped support for Python 2.5. See https://www.zyte.com/blog/scrapy-0-15-dropping-support-for-python-2-5/
-- dropped support for Twisted 2.5
-- added :setting:`REFERER_ENABLED` setting, to control referer middleware
-- changed default user agent to: ``Scrapy/VERSION (+http://scrapy.org)``
-- removed (undocumented) ``HTMLImageLinkExtractor`` class from ``scrapy.contrib.linkextractors.image``
-- removed per-spider settings (to be replaced by instantiating multiple crawler objects)
-- ``USER_AGENT`` spider attribute will no longer work, use ``user_agent`` attribute instead
-- ``DOWNLOAD_TIMEOUT`` spider attribute will no longer work, use ``download_timeout`` attribute instead
-- removed ``ENCODING_ALIASES`` setting, as encoding auto-detection has been moved to the `w3lib`_ library
-- promoted :ref:`topics-djangoitem` to main contrib
-- LogFormatter method now return dicts(instead of strings) to support lazy formatting (:gh:`164`, :commit:`dcef7b0`)
-- downloader handlers (:setting:`DOWNLOAD_HANDLERS` setting) now receive settings as the first argument of the ``__init__`` method
-- replaced memory usage accounting with (more portable) `resource`_ module, removed ``scrapy.utils.memory`` module
-- removed signal: ``scrapy.mail.mail_sent``
-- removed ``TRACK_REFS`` setting, now :ref:`trackrefs <topics-leaks-trackrefs>` is always enabled
-- DBM is now the default storage backend for HTTP cache middleware
-- number of log messages (per level) are now tracked through Scrapy stats (stat name: ``log_count/LEVEL``)
-- number received responses are now tracked through Scrapy stats (stat name: ``response_received_count``)
-- removed ``scrapy.log.started`` attribute
-
-Scrapy 0.14.4
--------------
-
-- added precise to supported Ubuntu distros (:commit:`b7e46df`)
-- fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py (:commit:`340fbdb`)
-- meta tag attributes for content-type http equiv can be in any order. #123 (:commit:`0cb68af`)
-- replace "import Image" by more standard "from PIL import Image". closes #88 (:commit:`4d17048`)
-- return trial status as bin/runtests.sh exit value. #118 (:commit:`b7b2e7f`)
-
-Scrapy 0.14.3
--------------
-
-- forgot to include pydispatch license. #118 (:commit:`fd85f9c`)
-- include egg files used by testsuite in source distribution. #118 (:commit:`c897793`)
-- update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (:commit:`2548dcc`)
-- added note to docs/topics/firebug.rst about google directory being shut down (:commit:`668e352`)
-- don't discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)
-- do not fail handling unicode xpaths in libxml2 backed selectors (:commit:`b830e95`)
-- fixed minor mistake in Request objects documentation (:commit:`bf3c9ee`)
-- fixed minor defect in link extractors documentation (:commit:`ba14f38`)
-- removed some obsolete remaining code related to sqlite support in Scrapy (:commit:`0665175`)
-
-Scrapy 0.14.2
--------------
-
-- move buffer pointing to start of file before computing checksum. refs #92 (:commit:`6a5bef2`)
-- Compute image checksum before persisting images. closes #92 (:commit:`9817df1`)
-- remove leaking references in cached failures (:commit:`673a120`)
-- fixed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) (:commit:`11133e9`)
-- fixed struct.error on http compression middleware. closes #87 (:commit:`1423140`)
-- ajax crawling wasn't expanding for unicode urls (:commit:`0de3fb4`)
-- Catch ``start_requests()`` iterator errors. refs #83 (:commit:`454a21d`)
-- Speed-up libxml2 XPathSelector (:commit:`2fbd662`)
-- updated versioning doc according to recent changes (:commit:`0a070f5`)
-- scrapyd: fixed documentation link (:commit:`2b4e4c3`)
-- extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`)
-
-Scrapy 0.14.1
--------------
-
-- extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`)
-- bumped version to 0.14.1 (:commit:`6cb9e1c`)
-- fixed reference to tutorial directory (:commit:`4b86bd6`)
-- doc: removed duplicated callback argument from Request.replace() (:commit:`1aeccdd`)
-- fixed formatting of scrapyd doc (:commit:`8bf19e6`)
-- Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension (:commit:`14a8e6e`)
-- added comment about why we disable ssl on boto images upload (:commit:`5223575`)
-- SSL handshaking hangs when doing too many parallel connections to S3 (:commit:`63d583d`)
-- change tutorial to follow changes on dmoz site (:commit:`bcb3198`)
-- Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 (:commit:`98f3f87`)
-- allow spider to set autothrottle max concurrency (:commit:`175a4b5`)
-
-Scrapy 0.14
------------
-
-New features and settings
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Support for AJAX crawlable urls
-- New persistent scheduler that stores requests on disk, allowing to suspend and resume crawls (:rev:`2737`)
-- added ``-o`` option to ``scrapy crawl``, a shortcut for dumping scraped items into a file (or standard output using ``-``)
-- Added support for passing custom settings to Scrapyd ``schedule.json`` api (:rev:`2779`, :rev:`2783`)
-- New ``ChunkedTransferMiddleware`` (enabled by default) to support `chunked transfer encoding`_ (:rev:`2769`)
-- Add boto 2.0 support for S3 downloader handler (:rev:`2763`)
-- Added `marshal`_ to formats supported by feed exports (:rev:`2744`)
-- In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)
-- Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)
- - ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:
- - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``
- - check the documentation for more details
-- Added builtin caching DNS resolver (:rev:`2728`)
-- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)
-- Moved spider queues to scrapyd: ``scrapy.spiderqueue`` -> ``scrapyd.spiderqueue`` (:rev:`2708`)
-- Moved sqlite utils to scrapyd: ``scrapy.utils.sqlite`` -> ``scrapyd.sqlite`` (:rev:`2781`)
-- Real support for returning iterators on ``start_requests()`` method. The iterator is now consumed during the crawl when the spider is getting idle (:rev:`2704`)
-- Added :setting:`REDIRECT_ENABLED` setting to quickly enable/disable the redirect middleware (:rev:`2697`)
-- Added :setting:`RETRY_ENABLED` setting to quickly enable/disable the retry middleware (:rev:`2694`)
-- Added ``CloseSpider`` exception to manually close spiders (:rev:`2691`)
-- Improved encoding detection by adding support for HTML5 meta charset declaration (:rev:`2690`)
-- Refactored close spider behavior to wait for all downloads to finish and be processed by spiders, before closing the spider (:rev:`2688`)
-- Added ``SitemapSpider`` (see documentation in Spiders page) (:rev:`2658`)
-- Added ``LogStats`` extension for periodically logging basic stats (like crawled pages and scraped items) (:rev:`2657`)
-- Make handling of gzipped responses more robust (#319, :rev:`2643`). Now Scrapy will try and decompress as much as possible from a gzipped response, instead of failing with an ``IOError``.
-- Simplified !MemoryDebugger extension to use stats for dumping memory debugging info (:rev:`2639`)
-- Added new command to edit spiders: ``scrapy edit`` (:rev:`2636`) and ``-e`` flag to ``genspider`` command that uses it (:rev:`2653`)
-- Changed default representation of items to pretty-printed dicts. (:rev:`2631`). This improves default logging by making log more readable in the default case, for both Scraped and Dropped lines.
-- Added :signal:`spider_error` signal (:rev:`2628`)
-- Added :setting:`COOKIES_ENABLED` setting (:rev:`2625`)
-- Stats are now dumped to Scrapy log (default value of :setting:`STATS_DUMP` setting has been changed to ``True``). This is to make Scrapy users more aware of Scrapy stats and the data that is collected there.
-- Added support for dynamically adjusting download delay and maximum concurrent requests (:rev:`2599`)
-- Added new DBM HTTP cache storage backend (:rev:`2576`)
-- Added ``listjobs.json`` API to Scrapyd (:rev:`2571`)
-- ``CsvItemExporter``: added ``join_multivalued`` parameter (:rev:`2578`)
-- Added namespace support to ``xmliter_lxml`` (:rev:`2552`)
-- Improved cookies middleware by making ``COOKIES_DEBUG`` nicer and documenting it (:rev:`2579`)
-- Several improvements to Scrapyd and Link extractors
-
-Code rearranged and removed
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Merged item passed and item scraped concepts, as they have often proved confusing in the past. This means: (:rev:`2630`)
- - original item_scraped signal was removed
- - original item_passed signal was renamed to item_scraped
- - old log lines ``Scraped Item...`` were removed
- - old log lines ``Passed Item...`` were renamed to ``Scraped Item...`` lines and downgraded to ``DEBUG`` level
-- Reduced Scrapy codebase by striping part of Scrapy code into two new libraries:
- - `w3lib`_ (several functions from ``scrapy.utils.{http,markup,multipart,response,url}``, done in :rev:`2584`)
- - `scrapely`_ (was ``scrapy.contrib.ibl``, done in :rev:`2586`)
-- Removed unused function: ``scrapy.utils.request.request_info()`` (:rev:`2577`)
-- Removed googledir project from ``examples/googledir``. There's now a new example project called ``dirbot`` available on GitHub: https://github.com/scrapy/dirbot
-- Removed support for default field values in Scrapy items (:rev:`2616`)
-- Removed experimental crawlspider v2 (:rev:`2632`)
-- Removed scheduler middleware to simplify architecture. Duplicates filter is now done in the scheduler itself, using the same dupe filtering class as before (``DUPEFILTER_CLASS`` setting) (:rev:`2640`)
-- Removed support for passing urls to ``scrapy crawl`` command (use ``scrapy parse`` instead) (:rev:`2704`)
-- Removed deprecated Execution Queue (:rev:`2704`)
-- Removed (undocumented) spider context extension (from scrapy.contrib.spidercontext) (:rev:`2780`)
-- removed ``CONCURRENT_SPIDERS`` setting (use scrapyd maxproc instead) (:rev:`2789`)
-- Renamed attributes of core components: downloader.sites -> downloader.slots, scraper.sites -> scraper.slots (:rev:`2717`, :rev:`2718`)
-- Renamed setting ``CLOSESPIDER_ITEMPASSED`` to :setting:`CLOSESPIDER_ITEMCOUNT` (:rev:`2655`). Backward compatibility kept.
-
-Scrapy 0.12
------------
-
-The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
-
-New features and improvements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Passed item is now sent in the ``item`` argument of the :signal:`item_passed
- <item_scraped>` (#273)
-- Added verbose option to ``scrapy version`` command, useful for bug reports (#298)
-- HTTP cache now stored by default in the project data dir (#279)
-- Added project data storage directory (#276, #277)
-- Documented file structure of Scrapy projects (see command-line tool doc)
-- New lxml backend for XPath selectors (#147)
-- Per-spider settings (#245)
-- Support exit codes to signal errors in Scrapy commands (#248)
-- Added ``-c`` argument to ``scrapy shell`` command
-- Made ``libxml2`` optional (#260)
-- New ``deploy`` command (#261)
-- Added :setting:`CLOSESPIDER_PAGECOUNT` setting (#253)
-- Added :setting:`CLOSESPIDER_ERRORCOUNT` setting (#254)
-
-Scrapyd changes
-~~~~~~~~~~~~~~~
-
-- Scrapyd now uses one process per spider
-- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default)
-- A minimal web ui was added, available at http://localhost:6800 by default
-- There is now a ``scrapy server`` command to start a Scrapyd server of the current project
-
-Changes to settings
-~~~~~~~~~~~~~~~~~~~
-
-- added ``HTTPCACHE_ENABLED`` setting (False by default) to enable HTTP cache middleware
-- changed ``HTTPCACHE_EXPIRATION_SECS`` semantics: now zero means "never expire".
-
-Deprecated/obsoleted functionality
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Deprecated ``runserver`` command in favor of ``server`` command which starts a Scrapyd server. See also: Scrapyd changes
-- Deprecated ``queue`` command in favor of using Scrapyd ``schedule.json`` API. See also: Scrapyd changes
-- Removed the !LxmlItemLoader (experimental contrib which never graduated to main contrib)
-
-Scrapy 0.10
------------
-
-The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
-
-New features and improvements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- New Scrapy service called ``scrapyd`` for deploying Scrapy crawlers in production (#218) (documentation available)
-- Simplified Images pipeline usage which doesn't require subclassing your own images pipeline now (#217)
-- Scrapy shell now shows the Scrapy log by default (#206)
-- Refactored execution queue in a common base code and pluggable backends called "spider queues" (#220)
-- New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders to run.
-- Added documentation for Scrapy command-line tool and all its available sub-commands. (documentation available)
-- Feed exporters with pluggable backends (#197) (documentation available)
-- Deferred signals (#193)
-- Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195)
-- Support for overriding default request headers per spider (#181)
-- Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186)
-- Split Debian package into two packages - the library and the service (#187)
-- Scrapy log refactoring (#188)
-- New extension for keeping persistent spider contexts among different runs (#203)
-- Added ``dont_redirect`` request.meta key for avoiding redirects (#233)
-- Added ``dont_retry`` request.meta key for avoiding retries (#234)
-
-Command-line tool changes
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- New ``scrapy`` command which replaces the old ``scrapy-ctl.py`` (#199)
- - there is only one global ``scrapy`` command now, instead of one ``scrapy-ctl.py`` per project
- - Added ``scrapy.bat`` script for running more conveniently from Windows
-- Added bash completion to command-line tool (#210)
-- Renamed command ``start`` to ``runserver`` (#209)
-
-API changes
-~~~~~~~~~~~
-
-- ``url`` and ``body`` attributes of Request objects are now read-only (#230)
-- ``Request.copy()`` and ``Request.replace()`` now also copies their ``callback`` and ``errback`` attributes (#231)
-- Removed ``UrlFilterMiddleware`` from ``scrapy.contrib`` (already disabled by default)
-- Offsite middleware doesn't filter out any request coming from a spider that doesn't have a allowed_domains attribute (#225)
-- Removed Spider Manager ``load()`` method. Now spiders are loaded in the ``__init__`` method itself.
-- Changes to Scrapy Manager (now called "Crawler"):
- - ``scrapy.core.manager.ScrapyManager`` class renamed to ``scrapy.crawler.Crawler``
- - ``scrapy.core.manager.scrapymanager`` singleton moved to ``scrapy.project.crawler``
-- Moved module: ``scrapy.contrib.spidermanager`` to ``scrapy.spidermanager``
-- Spider Manager singleton moved from ``scrapy.spider.spiders`` to the ``spiders`` attribute of ``scrapy.project.crawler`` singleton.
-- moved Stats Collector classes: (#204)
- - ``scrapy.stats.collector.StatsCollector`` to ``scrapy.statscol.StatsCollector``
- - ``scrapy.stats.collector.SimpledbStatsCollector`` to ``scrapy.contrib.statscol.SimpledbStatsCollector``
-- default per-command settings are now specified in the ``default_settings`` attribute of command object class (#201)
-- changed arguments of Item pipeline ``process_item()`` method from ``(spider, item)`` to ``(item, spider)``
- - backward compatibility kept (with deprecation warning)
-- moved ``scrapy.core.signals`` module to ``scrapy.signals``
- - backward compatibility kept (with deprecation warning)
-- moved ``scrapy.core.exceptions`` module to ``scrapy.exceptions``
- - backward compatibility kept (with deprecation warning)
-- added ``handles_request()`` class method to ``BaseSpider``
-- dropped ``scrapy.log.exc()`` function (use ``scrapy.log.err()`` instead)
-- dropped ``component`` argument of ``scrapy.log.msg()`` function
-- dropped ``scrapy.log.log_level`` attribute
-- Added ``from_settings()`` class methods to Spider Manager, and Item Pipeline Manager
-
-Changes to settings
-~~~~~~~~~~~~~~~~~~~
-
-- Added ``HTTPCACHE_IGNORE_SCHEMES`` setting to ignore certain schemes on !HttpCacheMiddleware (#225)
-- Added ``SPIDER_QUEUE_CLASS`` setting which defines the spider queue to use (#220)
-- Added ``KEEP_ALIVE`` setting (#220)
-- Removed ``SERVICE_QUEUE`` setting (#220)
-- Removed ``COMMANDS_SETTINGS_MODULE`` setting (#201)
-- Renamed ``REQUEST_HANDLERS`` to ``DOWNLOAD_HANDLERS`` and make download handlers classes (instead of functions)
-
-Scrapy 0.9
-----------
-
-The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
-
-New features and improvements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Added SMTP-AUTH support to scrapy.mail
-- New settings added: ``MAIL_USER``, ``MAIL_PASS`` (:rev:`2065` | #149)
-- Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy (:rev:`2039`)
-- Added web service for controlling Scrapy process (this also deprecates the web console. (:rev:`2053` | #167)
-- Support for running Scrapy as a service, for production systems (:rev:`1988`, :rev:`2054`, :rev:`2055`, :rev:`2056`, :rev:`2057` | #168)
-- Added wrapper induction library (documentation only available in source code for now). (:rev:`2011`)
-- Simplified and improved response encoding support (:rev:`1961`, :rev:`1969`)
-- Added ``LOG_ENCODING`` setting (:rev:`1956`, documentation available)
-- Added ``RANDOMIZE_DOWNLOAD_DELAY`` setting (enabled by default) (:rev:`1923`, doc available)
-- ``MailSender`` is no longer IO-blocking (:rev:`1955` | #146)
-- Linkextractors and new Crawlspider now handle relative base tag urls (:rev:`1960` | #148)
-- Several improvements to Item Loaders and processors (:rev:`2022`, :rev:`2023`, :rev:`2024`, :rev:`2025`, :rev:`2026`, :rev:`2027`, :rev:`2028`, :rev:`2029`, :rev:`2030`)
-- Added support for adding variables to telnet console (:rev:`2047` | #165)
-- Support for requests without callbacks (:rev:`2050` | #166)
-
-API changes
-~~~~~~~~~~~
-
-- Change ``Spider.domain_name`` to ``Spider.name`` (SEP-012, :rev:`1975`)
-- ``Response.encoding`` is now the detected encoding (:rev:`1961`)
-- ``HttpErrorMiddleware`` now returns None or raises an exception (:rev:`2006` | #157)
-- ``scrapy.command`` modules relocation (:rev:`2035`, :rev:`2036`, :rev:`2037`)
-- Added ``ExecutionQueue`` for feeding spiders to scrape (:rev:`2034`)
-- Removed ``ExecutionEngine`` singleton (:rev:`2039`)
-- Ported ``S3ImagesStore`` (images pipeline) to use boto and threads (:rev:`2033`)
-- Moved module: ``scrapy.management.telnet`` to ``scrapy.telnet`` (:rev:`2047`)
-
-Changes to default settings
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-- Changed default ``SCHEDULER_ORDER`` to ``DFO`` (:rev:`1939`)
-
-Scrapy 0.8
-----------
-
-The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
-
-New features
-~~~~~~~~~~~~
-
-- Added DEFAULT_RESPONSE_ENCODING setting (:rev:`1809`)
-- Added ``dont_click`` argument to ``FormRequest.from_response()`` method (:rev:`1813`, :rev:`1816`)
-- Added ``clickdata`` argument to ``FormRequest.from_response()`` method (:rev:`1802`, :rev:`1803`)
-- Added support for HTTP proxies (``HttpProxyMiddleware``) (:rev:`1781`, :rev:`1785`)
-- Offsite spider middleware now logs messages when filtering out requests (:rev:`1841`)
-
-Backward-incompatible changes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. toctree::
+ :maxdepth: 1
-- Changed ``scrapy.utils.response.get_meta_refresh()`` signature (:rev:`1804`)
-- Removed deprecated ``scrapy.item.ScrapedItem`` class - use ``scrapy.item.Item instead`` (:rev:`1838`)
-- Removed deprecated ``scrapy.xpath`` module - use ``scrapy.selector`` instead. (:rev:`1836`)
-- Removed deprecated ``core.signals.domain_open`` signal - use ``core.signals.domain_opened`` instead (:rev:`1822`)
-- ``log.msg()`` now receives a ``spider`` argument (:rev:`1822`)
- - Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the ``spider`` argument and pass spider references. If you really want to pass a string, use the ``component`` argument instead.
-- Changed core signals ``domain_opened``, ``domain_closed``, ``domain_idle``
-- Changed Item pipeline to use spiders instead of domains
- - The ``domain`` argument of ``process_item()`` item pipeline method was changed to ``spider``, the new signature is: ``process_item(spider, item)`` (:rev:`1827` | #105)
- - To quickly port your code (to work with Scrapy 0.8) just use ``spider.domain_name`` where you previously used ``domain``.
-- Changed Stats API to use spiders instead of domains (:rev:`1849` | #113)
- - ``StatsCollector`` was changed to receive spider references (instead of domains) in its methods (``set_value``, ``inc_value``, etc).
- - added ``StatsCollector.iter_spider_stats()`` method
- - removed ``StatsCollector.list_domains()`` method
- - Also, Stats signals were renamed and now pass around spider references (instead of domains). Here's a summary of the changes:
- - To quickly port your code (to work with Scrapy 0.8) just use ``spider.domain_name`` where you previously used ``domain``. ``spider_stats`` contains exactly the same data as ``domain_stats``.
-- ``CloseDomain`` extension moved to ``scrapy.contrib.closespider.CloseSpider`` (:rev:`1833`)
- - Its settings were also renamed:
- - ``CLOSEDOMAIN_TIMEOUT`` to ``CLOSESPIDER_TIMEOUT``
- - ``CLOSEDOMAIN_ITEMCOUNT`` to ``CLOSESPIDER_ITEMCOUNT``
-- Removed deprecated ``SCRAPYSETTINGS_MODULE`` environment variable - use ``SCRAPY_SETTINGS_MODULE`` instead (:rev:`1840`)
-- Renamed setting: ``REQUESTS_PER_DOMAIN`` to ``CONCURRENT_REQUESTS_PER_SPIDER`` (:rev:`1830`, :rev:`1844`)
-- Renamed setting: ``CONCURRENT_DOMAINS`` to ``CONCURRENT_SPIDERS`` (:rev:`1830`)
-- Refactored HTTP Cache middleware
-- HTTP Cache middleware has been heavily refactored, retaining the same functionality except for the domain sectorization which was removed. (:rev:`1843` )
-- Renamed exception: ``DontCloseDomain`` to ``DontCloseSpider`` (:rev:`1859` | #120)
-- Renamed extension: ``DelayedCloseDomain`` to ``SpiderCloseDelay`` (:rev:`1861` | #121)
-- Removed obsolete ``scrapy.utils.markup.remove_escape_chars`` function - use ``scrapy.utils.markup.replace_escape_chars`` instead (:rev:`1865`)
-
-Scrapy 0.7
-----------
-
-First release of Scrapy.
+ news/2.x
+ news/1.x
+ news/0.x
.. _aiohttp: https://docs.aiohttp.org/en/stable/
-.. _boto3: https://github.com/boto/boto3
.. _botocore: https://github.com/boto/botocore
.. _charset-normalizer: https://charset-normalizer.readthedocs.io/en/latest/
-.. _chunked transfer encoding: https://en.wikipedia.org/wiki/Chunked_transfer_encoding
-.. _ClientForm: https://pypi.org/project/ClientForm/
-.. _colorlog: https://github.com/borntyping/python-colorlog
-.. _Creating a pull request: https://help.github.com/en/articles/creating-a-pull-request
.. _cryptography: https://cryptography.io/en/latest/
-.. _docstrings: https://docs.python.org/3/glossary.html#term-docstring
.. _ipdb: https://github.com/gotcha/ipdb
-.. _KeyboardInterrupt: https://docs.python.org/3/library/exceptions.html#KeyboardInterrupt
-.. _LevelDB: https://github.com/google/leveldb
.. _lxml: https://lxml.de/
-.. _marshal: https://docs.python.org/2/library/marshal.html
.. _parso: https://github.com/davidhalter/parso
-.. _parsel: https://github.com/scrapy/parsel
-.. _parsel.csstranslator.GenericTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.GenericTranslator
-.. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator
-.. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr
.. _platformdirs: https://platformdirs.readthedocs.io/en/latest/
-.. _PEP 257: https://peps.python.org/pep-0257/
.. _Pillow: https://github.com/python-pillow/Pillow
.. _pyOpenSSL: https://www.pyopenssl.org/en/stable/
.. _queuelib: https://github.com/scrapy/queuelib
-.. _registered with IANA: https://www.iana.org/assignments/media-types/media-types.xhtml
-.. _resource: https://docs.python.org/2/library/resource.html
-.. _robots.txt: https://www.robotstxt.org/
-.. _scrapely: https://github.com/scrapy/scrapely
-.. _scrapy-bench: https://github.com/scrapy/scrapy-bench
.. _service_identity: https://service-identity.readthedocs.io/en/stable/
-.. _six: https://six.readthedocs.io/
-.. _tox: https://pypi.org/project/tox/
-.. _Twisted: https://twisted.org/
.. _w3lib: https://github.com/scrapy/w3lib
-.. _w3lib.encoding: https://github.com/scrapy/w3lib/blob/master/w3lib/encoding.py
-.. _What is cacheable: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1
.. _zope.interface: https://zopeinterface.readthedocs.io/en/latest/
-.. _Zsh: https://www.zsh.org/
-.. _zstandard: https://pypi.org/project/zstandard/
diff --git a/docs/news/0.x.rst b/docs/news/0.x.rst
new file mode 100644
index 000000000..bc8c88348
--- /dev/null
+++ b/docs/news/0.x.rst
@@ -0,0 +1,923 @@
+.. _news-0.x:
+
+Scrapy 0.x release notes
+========================
+
+Scrapy 0.24.6 (2015-04-20)
+--------------------------
+
+- encode invalid xpath with unicode_escape under PY2 (:commit:`07cb3e5`)
+- fix IPython shell scope issue and load IPython user config (:commit:`2c8e573`)
+- Fix small typo in the docs (:commit:`d694019`)
+- Fix small typo (:commit:`f92fa83`)
+- Converted sel.xpath() calls to response.xpath() in Extracting the data (:commit:`c2c6d15`)
+
+
+Scrapy 0.24.5 (2015-02-25)
+--------------------------
+
+- Support new _getEndpoint Agent signatures on Twisted 15.0.0 (:commit:`540b9bc`)
+- DOC a couple more references are fixed (:commit:`b4c454b`)
+- DOC fix a reference (:commit:`e3c1260`)
+- t.i.b.ThreadedResolver is now a new-style class (:commit:`9e13f42`)
+- S3DownloadHandler: fix auth for requests with quoted paths/query params (:commit:`cdb9a0b`)
+- fixed the variable types in mailsender documentation (:commit:`bb3a848`)
+- Reset items_scraped instead of item_count (:commit:`edb07a4`)
+- Tentative attention message about what document to read for contributions (:commit:`7ee6f7a`)
+- mitmproxy 0.10.1 needs netlib 0.10.1 too (:commit:`874fcdd`)
+- pin mitmproxy 0.10.1 as >0.11 does not work with tests (:commit:`c6b21f0`)
+- Test the parse command locally instead of against an external url (:commit:`c3a6628`)
+- Patches Twisted issue while closing the connection pool on HTTPDownloadHandler (:commit:`d0bf957`)
+- Updates documentation on dynamic item classes. (:commit:`eeb589a`)
+- Merge pull request #943 from Lazar-T/patch-3 (:commit:`5fdab02`)
+- typo (:commit:`b0ae199`)
+- pywin32 is required by Twisted. closes #937 (:commit:`5cb0cfb`)
+- Update install.rst (:commit:`781286b`)
+- Merge pull request #928 from Lazar-T/patch-1 (:commit:`b415d04`)
+- comma instead of fullstop (:commit:`627b9ba`)
+- Merge pull request #885 from jsma/patch-1 (:commit:`de909ad`)
+- Update request-response.rst (:commit:`3f3263d`)
+- SgmlLinkExtractor - fix for parsing <area> tag with Unicode present (:commit:`49b40f0`)
+
+Scrapy 0.24.4 (2014-08-09)
+--------------------------
+
+- pem file is used by mockserver and required by scrapy bench (:commit:`5eddc68b63`)
+- scrapy bench needs scrapy.tests* (:commit:`d6cb999`)
+
+Scrapy 0.24.3 (2014-08-09)
+--------------------------
+
+- no need to waste travis-ci time on py3 for 0.24 (:commit:`8e080c1`)
+- Update installation docs (:commit:`1d0c096`)
+- There is a trove classifier for Scrapy framework! (:commit:`4c701d7`)
+- update other places where w3lib version is mentioned (:commit:`d109c13`)
+- Update w3lib requirement to 1.8.0 (:commit:`39d2ce5`)
+- Use w3lib.html.replace_entities() (remove_entities() is deprecated) (:commit:`180d3ad`)
+- set zip_safe=False (:commit:`a51ee8b`)
+- do not ship tests package (:commit:`ee3b371`)
+- scrapy.bat is not needed anymore (:commit:`c3861cf`)
+- Modernize setup.py (:commit:`362e322`)
+- headers can not handle non-string values (:commit:`94a5c65`)
+- fix ftp test cases (:commit:`a274a7f`)
+- The sum up of travis-ci builds are taking like 50min to complete (:commit:`ae1e2cc`)
+- Update shell.rst typo (:commit:`e49c96a`)
+- removes weird indentation in the shell results (:commit:`1ca489d`)
+- improved explanations, clarified blog post as source, added link for XPath string functions in the spec (:commit:`65c8f05`)
+- renamed UserTimeoutError and ServerTimeouterror #583 (:commit:`037f6ab`)
+- adding some xpath tips to selectors docs (:commit:`2d103e0`)
+- fix tests to account for https://github.com/scrapy/w3lib/pull/23 (:commit:`f8d366a`)
+- get_func_args maximum recursion fix #728 (:commit:`81344ea`)
+- Updated input/output processor example according to #560. (:commit:`f7c4ea8`)
+- Fixed Python syntax in tutorial. (:commit:`db59ed9`)
+- Add test case for tunneling proxy (:commit:`f090260`)
+- Bugfix for leaking Proxy-Authorization header to remote host when using tunneling (:commit:`d8793af`)
+- Extract links from XHTML documents with MIME-Type "application/xml" (:commit:`ed1f376`)
+- Merge pull request #793 from roysc/patch-1 (:commit:`91a1106`)
+- Fix typo in commands.rst (:commit:`743e1e2`)
+- better testcase for settings.overrides.setdefault (:commit:`e22daaf`)
+- Using CRLF as line marker according to http 1.1 definition (:commit:`5ec430b`)
+
+Scrapy 0.24.2 (2014-07-08)
+--------------------------
+
+- Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute (:commit:`e5e8133`)
+- there is not support for python3 yet (:commit:`3cd6146`)
+- Update python compatible version set to Debian packages (:commit:`fa5d76b`)
+- DOC fix formatting in release notes (:commit:`c6a9e20`)
+
+Scrapy 0.24.1 (2014-06-27)
+--------------------------
+
+- Fix deprecated CrawlerSettings and increase backward compatibility with
+ .defaults attribute (:commit:`8e3f20a`)
+
+
+Scrapy 0.24.0 (2014-06-26)
+--------------------------
+
+Enhancements
+~~~~~~~~~~~~
+
+- Improve Scrapy top-level namespace (:gh:`494`, :gh:`684`)
+- Add selector shortcuts to responses (:gh:`554`, :gh:`690`)
+- Add new lxml based LinkExtractor to replace unmaintained SgmlLinkExtractor
+ (:gh:`559`, :gh:`761`, :gh:`763`)
+- Cleanup settings API - part of per-spider settings **GSoC project** (:gh:`737`)
+- Add UTF8 encoding header to templates (:gh:`688`, :gh:`762`)
+- Telnet console now binds to 127.0.0.1 by default (:gh:`699`)
+- Update Debian/Ubuntu install instructions (:gh:`509`, :gh:`549`)
+- Disable smart strings in lxml XPath evaluations (:gh:`535`)
+- Restore filesystem based cache as default for http
+ cache middleware (:gh:`541`, :gh:`500`, :gh:`571`)
+- Expose current crawler in Scrapy shell (:gh:`557`)
+- Improve testsuite comparing CSV and XML exporters (:gh:`570`)
+- New ``offsite/filtered`` and ``offsite/domains`` stats (:gh:`566`)
+- Support process_links as generator in CrawlSpider (:gh:`555`)
+- Verbose logging and new stats counters for DupeFilter (:gh:`553`)
+- Add a mimetype parameter to ``MailSender.send()`` (:gh:`602`)
+- Generalize file pipeline log messages (:gh:`622`)
+- Replace unencodeable codepoints with html entities in SGMLLinkExtractor (:gh:`565`)
+- Converted SEP documents to rst format (:gh:`629`, :gh:`630`,
+ :gh:`638`, :gh:`632`, :gh:`636`, :gh:`640`, :gh:`635`,
+ :gh:`634`, :gh:`639`, :gh:`637`, :gh:`631`, :gh:`633`,
+ :gh:`641`, :gh:`642`)
+- Tests and docs for clickdata's nr index in FormRequest (:gh:`646`, :gh:`645`)
+- Allow to disable a downloader handler just like any other component (:gh:`650`)
+- Log when a request is discarded after too many redirections (:gh:`654`)
+- Log error responses if they are not handled by spider callbacks
+ (:gh:`612`, :gh:`656`)
+- Add content-type check to http compression mw (:gh:`193`, :gh:`660`)
+- Run pypy tests using latest pypi from ppa (:gh:`674`)
+- Run test suite using pytest instead of trial (:gh:`679`)
+- Build docs and check for dead links in tox environment (:gh:`687`)
+- Make scrapy.version_info a tuple of integers (:gh:`681`, :gh:`692`)
+- Infer exporter's output format from filename extensions
+ (:gh:`546`, :gh:`659`, :gh:`760`)
+- Support case-insensitive domains in ``url_is_from_any_domain()`` (:gh:`693`)
+- Remove pep8 warnings in project and spider templates (:gh:`698`)
+- Tests and docs for ``request_fingerprint`` function (:gh:`597`)
+- Update SEP-19 for GSoC project ``per-spider settings`` (:gh:`705`)
+- Set exit code to non-zero when contracts fails (:gh:`727`)
+- Add a setting to control what class is instantiated as Downloader component
+ (:gh:`738`)
+- Pass response in ``item_dropped`` signal (:gh:`724`)
+- Improve ``scrapy check`` contracts command (:gh:`733`, :gh:`752`)
+- Document ``spider.closed()`` shortcut (:gh:`719`)
+- Document ``request_scheduled`` signal (:gh:`746`)
+- Add a note about reporting security issues (:gh:`697`)
+- Add LevelDB http cache storage backend (:gh:`626`, :gh:`500`)
+- Sort spider list output of ``scrapy list`` command (:gh:`742`)
+- Multiple documentation enhancements and fixes
+ (:gh:`575`, :gh:`587`, :gh:`590`, :gh:`596`, :gh:`610`,
+ :gh:`617`, :gh:`618`, :gh:`627`, :gh:`613`, :gh:`643`,
+ :gh:`654`, :gh:`675`, :gh:`663`, :gh:`711`, :gh:`714`)
+
+Bugfixes
+~~~~~~~~
+
+- Encode unicode URL value when creating Links in RegexLinkExtractor (:gh:`561`)
+- Ignore None values in ItemLoader processors (:gh:`556`)
+- Fix link text when there is an inner tag in SGMLLinkExtractor and
+ HtmlParserLinkExtractor (:gh:`485`, :gh:`574`)
+- Fix wrong checks on subclassing of deprecated classes
+ (:gh:`581`, :gh:`584`)
+- Handle errors caused by inspect.stack() failures (:gh:`582`)
+- Fix a reference to unexistent engine attribute (:gh:`593`, :gh:`594`)
+- Fix dynamic itemclass example usage of type() (:gh:`603`)
+- Use lucasdemarchi/codespell to fix typos (:gh:`628`)
+- Fix default value of attrs argument in SgmlLinkExtractor to be tuple (:gh:`661`)
+- Fix XXE flaw in sitemap reader (:gh:`676`)
+- Fix engine to support filtered start requests (:gh:`707`)
+- Fix offsite middleware case on urls with no hostnames (:gh:`745`)
+- Testsuite doesn't require PIL anymore (:gh:`585`)
+
+
+Scrapy 0.22.2 (2014-02-14)
+--------------------------
+
+- fix a reference to unexistent engine.slots. closes #593 (:commit:`13c099a`)
+- downloaderMW doc typo (spiderMW doc copy remnant) (:commit:`8ae11bf`)
+- Correct typos (:commit:`1346037`)
+
+Scrapy 0.22.1 (2014-02-08)
+--------------------------
+
+- localhost666 can resolve under certain circumstances (:commit:`2ec2279`)
+- test inspect.stack failure (:commit:`cc3eda3`)
+- Handle cases when inspect.stack() fails (:commit:`8cb44f9`)
+- Fix wrong checks on subclassing of deprecated classes. closes #581 (:commit:`46d98d6`)
+- Docs: 4-space indent for final spider example (:commit:`13846de`)
+- Fix HtmlParserLinkExtractor and tests after #485 merge (:commit:`368a946`)
+- BaseSgmlLinkExtractor: Fixed the missing space when the link has an inner tag (:commit:`b566388`)
+- BaseSgmlLinkExtractor: Added unit test of a link with an inner tag (:commit:`c1cb418`)
+- BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag (:commit:`7e4d627`)
+- Fix tests for Travis-CI build (:commit:`76c7e20`)
+- replace unencodeable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)
+- RegexLinkExtractor: encode URL unicode value when creating Links (:commit:`d0ee545`)
+- Updated the tutorial crawl output with latest output. (:commit:`8da65de`)
+- Updated shell docs with the crawler reference and fixed the actual shell output. (:commit:`875b9ab`)
+- PEP8 minor edits. (:commit:`f89efaf`)
+- Expose current crawler in the Scrapy shell. (:commit:`5349cec`)
+- Unused re import and PEP8 minor edits. (:commit:`387f414`)
+- Ignore None's values when using the ItemLoader. (:commit:`0632546`)
+- DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. (:commit:`cde9a8c`)
+- show Ubuntu setup instructions as literal code (:commit:`fb5c9c5`)
+- Update Ubuntu installation instructions (:commit:`70fb105`)
+- Merge pull request #550 from stray-leone/patch-1 (:commit:`6f70b6a`)
+- modify the version of Scrapy Ubuntu package (:commit:`725900d`)
+- fix 0.22.0 release date (:commit:`af0219a`)
+- fix typos in news.rst and remove (not released yet) header (:commit:`b7f58f4`)
+
+Scrapy 0.22.0 (2014-01-17)
+--------------------------
+
+Enhancements
+~~~~~~~~~~~~
+
+- [**Backward incompatible**] Switched HTTPCacheMiddleware backend to filesystem (:gh:`541`)
+ To restore old backend set ``HTTPCACHE_STORAGE`` to ``scrapy.contrib.httpcache.DbmCacheStorage``
+- Proxy \https:// urls using CONNECT method (:gh:`392`, :gh:`397`)
+- Add a middleware to crawl ajax crawlable pages as defined by google (:gh:`343`)
+- Rename scrapy.spider.BaseSpider to scrapy.spider.Spider (:gh:`510`, :gh:`519`)
+- Selectors register EXSLT namespaces by default (:gh:`472`)
+- Unify item loaders similar to selectors renaming (:gh:`461`)
+- Make ``RFPDupeFilter`` class easily subclassable (:gh:`533`)
+- Improve test coverage and forthcoming Python 3 support (:gh:`525`)
+- Promote startup info on settings and middleware to INFO level (:gh:`520`)
+- Support partials in ``get_func_args`` util (:gh:`506`, :gh:`504`)
+- Allow running individual tests via tox (:gh:`503`)
+- Update extensions ignored by link extractors (:gh:`498`)
+- Add middleware methods to get files/images/thumbs paths (:gh:`490`)
+- Improve offsite middleware tests (:gh:`478`)
+- Add a way to skip default Referer header set by RefererMiddleware (:gh:`475`)
+- Do not send ``x-gzip`` in default ``Accept-Encoding`` header (:gh:`469`)
+- Support defining http error handling using settings (:gh:`466`)
+- Use modern python idioms wherever you find legacies (:gh:`497`)
+- Improve and correct documentation
+ (:gh:`527`, :gh:`524`, :gh:`521`, :gh:`517`, :gh:`512`, :gh:`505`,
+ :gh:`502`, :gh:`489`, :gh:`465`, :gh:`460`, :gh:`425`, :gh:`536`)
+
+Fixes
+~~~~~
+
+- Update Selector class imports in CrawlSpider template (:gh:`484`)
+- Fix unexistent reference to ``engine.slots`` (:gh:`464`)
+- Do not try to call ``body_as_unicode()`` on a non-TextResponse instance (:gh:`462`)
+- Warn when subclassing XPathItemLoader, previously it only warned on
+ instantiation. (:gh:`523`)
+- Warn when subclassing XPathSelector, previously it only warned on
+ instantiation. (:gh:`537`)
+- Multiple fixes to memory stats (:gh:`531`, :gh:`530`, :gh:`529`)
+- Fix overriding url in ``FormRequest.from_response()`` (:gh:`507`)
+- Fix tests runner under pip 1.5 (:gh:`513`)
+- Fix logging error when spider name is unicode (:gh:`479`)
+
+Scrapy 0.20.2 (2013-12-09)
+--------------------------
+
+- Update CrawlSpider Template with Selector changes (:commit:`6d1457d`)
+- fix method name in tutorial. closes GH-480 (:commit:`b4fc359`
+
+Scrapy 0.20.1 (2013-11-28)
+--------------------------
+
+- include_package_data is required to build wheels from published sources (:commit:`5ba1ad5`)
+- process_parallel was leaking the failures on its internal deferreds. closes #458 (:commit:`419a780`)
+
+Scrapy 0.20.0 (2013-11-08)
+--------------------------
+
+Enhancements
+~~~~~~~~~~~~
+
+- New Selector's API including CSS selectors (:gh:`395` and :gh:`426`),
+- Request/Response url/body attributes are now immutable
+ (modifying them had been deprecated for a long time)
+- :setting:`ITEM_PIPELINES` is now defined as a dict (instead of a list)
+- Sitemap spider can fetch alternate URLs (:gh:`360`)
+- ``Selector.remove_namespaces()`` now remove namespaces from element's attributes. (:gh:`416`)
+- Paved the road for Python 3.3+ (:gh:`435`, :gh:`436`, :gh:`431`, :gh:`452`)
+- New item exporter using native python types with nesting support (:gh:`366`)
+- Tune HTTP1.1 pool size so it matches concurrency defined by settings (:commit:`b43b5f575`)
+- scrapy.mail.MailSender now can connect over TLS or upgrade using STARTTLS (:gh:`327`)
+- New FilesPipeline with functionality factored out from ImagesPipeline (:gh:`370`, :gh:`409`)
+- Recommend Pillow instead of PIL for image handling (:gh:`317`)
+- Added Debian packages for Ubuntu Quantal and Raring (:commit:`86230c0`)
+- Mock server (used for tests) can listen for HTTPS requests (:gh:`410`)
+- Remove multi spider support from multiple core components
+ (:gh:`422`, :gh:`421`, :gh:`420`, :gh:`419`, :gh:`423`, :gh:`418`)
+- Travis-CI now tests Scrapy changes against development versions of ``w3lib`` and ``queuelib`` python packages.
+- Add pypy 2.1 to continuous integration tests (:commit:`ecfa7431`)
+- Pylinted, pep8 and removed old-style exceptions from source (:gh:`430`, :gh:`432`)
+- Use importlib for parametric imports (:gh:`445`)
+- Handle a regression introduced in Python 2.7.5 that affects XmlItemExporter (:gh:`372`)
+- Bugfix crawling shutdown on SIGINT (:gh:`450`)
+- Do not submit ``reset`` type inputs in FormRequest.from_response (:commit:`b326b87`)
+- Do not silence download errors when request errback raises an exception (:commit:`684cfc0`)
+
+Bugfixes
+~~~~~~~~
+
+- Fix tests under Django 1.6 (:commit:`b6bed44c`)
+- Lot of bugfixes to retry middleware under disconnections using HTTP 1.1 download handler
+- Fix inconsistencies among Twisted releases (:gh:`406`)
+- Fix Scrapy shell bugs (:gh:`418`, :gh:`407`)
+- Fix invalid variable name in setup.py (:gh:`429`)
+- Fix tutorial references (:gh:`387`)
+- Improve request-response docs (:gh:`391`)
+- Improve best practices docs (:gh:`399`, :gh:`400`, :gh:`401`, :gh:`402`)
+- Improve django integration docs (:gh:`404`)
+- Document ``bindaddress`` request meta (:commit:`37c24e01d7`)
+- Improve ``Request`` class documentation (:gh:`226`)
+
+Other
+~~~~~
+
+- Dropped Python 2.6 support (:gh:`448`)
+- Add :doc:`cssselect <cssselect:index>` python package as install dependency
+- Drop libxml2 and multi selector's backend support, `lxml`_ is required from now on.
+- Minimum Twisted version increased to 10.0.0, dropped Twisted 8.0 support.
+- Running test suite now requires ``mock`` python library (:gh:`390`)
+
+
+Thanks
+~~~~~~
+
+Thanks to everyone who contribute to this release!
+
+List of contributors sorted by number of commits::
+
+ 69 Daniel Graña <dangra@...>
+ 37 Pablo Hoffman <pablo@...>
+ 13 Mikhail Korobov <kmike84@...>
+ 9 Alex Cepoi <alex.cepoi@...>
+ 9 alexanderlukanin13 <alexander.lukanin.13@...>
+ 8 Rolando Espinoza La fuente <darkrho@...>
+ 8 Lukasz Biedrycki <lukasz.biedrycki@...>
+ 6 Nicolas Ramirez <nramirez.uy@...>
+ 3 Paul Tremberth <paul.tremberth@...>
+ 2 Martin Olveyra <molveyra@...>
+ 2 Stefan <misc@...>
+ 2 Rolando Espinoza <darkrho@...>
+ 2 Loren Davie <loren@...>
+ 2 irgmedeiros <irgmedeiros@...>
+ 1 Stefan Koch <taikano@...>
+ 1 Stefan <cct@...>
+ 1 scraperdragon <dragon@...>
+ 1 Kumara Tharmalingam <ktharmal@...>
+ 1 Francesco Piccinno <stack.box@...>
+ 1 Marcos Campal <duendex@...>
+ 1 Dragon Dave <dragon@...>
+ 1 Capi Etheriel <barraponto@...>
+ 1 cacovsky <amarquesferraz@...>
+ 1 Berend Iwema <berend@...>
+
+Scrapy 0.18.4 (2013-10-10)
+--------------------------
+
+- IPython refuses to update the namespace. fix #396 (:commit:`3d32c4f`)
+- Fix AlreadyCalledError replacing a request in shell command. closes #407 (:commit:`b1d8919`)
+- Fix ``start_requests()`` laziness and early hangs (:commit:`89faf52`)
+
+Scrapy 0.18.3 (2013-10-03)
+--------------------------
+
+- fix regression on lazy evaluation of start requests (:commit:`12693a5`)
+- forms: do not submit reset inputs (:commit:`e429f63`)
+- increase unittest timeouts to decrease travis false positive failures (:commit:`912202e`)
+- backport master fixes to json exporter (:commit:`cfc2d46`)
+- Fix permission and set umask before generating sdist tarball (:commit:`06149e0`)
+
+Scrapy 0.18.2 (2013-09-03)
+--------------------------
+
+- Backport ``scrapy check`` command fixes and backward compatible multi
+ crawler process(:gh:`339`)
+
+Scrapy 0.18.1 (2013-08-27)
+--------------------------
+
+- remove extra import added by cherry picked changes (:commit:`d20304e`)
+- fix crawling tests under twisted pre 11.0.0 (:commit:`1994f38`)
+- py26 can not format zero length fields {} (:commit:`abf756f`)
+- test PotentiaDataLoss errors on unbound responses (:commit:`b15470d`)
+- Treat responses without content-length or Transfer-Encoding as good responses (:commit:`c4bf324`)
+- do no include ResponseFailed if http11 handler is not enabled (:commit:`6cbe684`)
+- New HTTP client wraps connection lost in ResponseFailed exception. fix #373 (:commit:`1a20bba`)
+- limit travis-ci build matrix (:commit:`3b01bb8`)
+- Merge pull request #375 from peterarenot/patch-1 (:commit:`fa766d7`)
+- Fixed so it refers to the correct folder (:commit:`3283809`)
+- added Quantal & Raring to support Ubuntu releases (:commit:`1411923`)
+- fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 (:commit:`bb35ed0`)
+- fix XmlItemExporter in Python 2.7.4 and 2.7.5 (:commit:`de3e451`)
+- minor updates to 0.18 release notes (:commit:`c45e5f1`)
+- fix contributors list format (:commit:`0b60031`)
+
+Scrapy 0.18.0 (2013-08-09)
+--------------------------
+
+- Lot of improvements to testsuite run using Tox, including a way to test on pypi
+- Handle GET parameters for AJAX crawlable urls (:commit:`3fe2a32`)
+- Use lxml recover option to parse sitemaps (:gh:`347`)
+- Bugfix cookie merging by hostname and not by netloc (:gh:`352`)
+- Support disabling ``HttpCompressionMiddleware`` using a flag setting (:gh:`359`)
+- Support xml namespaces using ``iternodes`` parser in ``XMLFeedSpider`` (:gh:`12`)
+- Support ``dont_cache`` request meta flag (:gh:`19`)
+- Bugfix ``scrapy.utils.gz.gunzip`` broken by changes in python 2.7.4 (:commit:`4dc76e`)
+- Bugfix url encoding on ``SgmlLinkExtractor`` (:gh:`24`)
+- Bugfix ``TakeFirst`` processor shouldn't discard zero (0) value (:gh:`59`)
+- Support nested items in xml exporter (:gh:`66`)
+- Improve cookies handling performance (:gh:`77`)
+- Log dupe filtered requests once (:gh:`105`)
+- Split redirection middleware into status and meta based middlewares (:gh:`78`)
+- Use HTTP1.1 as default downloader handler (:gh:`109` and :gh:`318`)
+- Support xpath form selection on ``FormRequest.from_response`` (:gh:`185`)
+- Bugfix unicode decoding error on ``SgmlLinkExtractor`` (:gh:`199`)
+- Bugfix signal dispatching on pypi interpreter (:gh:`205`)
+- Improve request delay and concurrency handling (:gh:`206`)
+- Add RFC2616 cache policy to ``HttpCacheMiddleware`` (:gh:`212`)
+- Allow customization of messages logged by engine (:gh:`214`)
+- Multiples improvements to ``DjangoItem`` (:gh:`217`, :gh:`218`, :gh:`221`)
+- Extend Scrapy commands using setuptools entry points (:gh:`260`)
+- Allow spider ``allowed_domains`` value to be set/tuple (:gh:`261`)
+- Support ``settings.getdict`` (:gh:`269`)
+- Simplify internal ``scrapy.core.scraper`` slot handling (:gh:`271`)
+- Added ``Item.copy`` (:gh:`290`)
+- Collect idle downloader slots (:gh:`297`)
+- Add ``ftp://`` scheme downloader handler (:gh:`329`)
+- Added downloader benchmark webserver and spider tools :ref:`benchmarking`
+- Moved persistent (on disk) queues to a separate project (queuelib_) which Scrapy now depends on
+- Add Scrapy commands using external libraries (:gh:`260`)
+- Added ``--pdb`` option to ``scrapy`` command line tool
+- Added :meth:`XPathSelector.remove_namespaces <scrapy.Selector.remove_namespaces>` which allows to remove all namespaces from XML documents for convenience (to work with namespace-less XPaths). Documented in :ref:`topics-selectors`.
+- Several improvements to spider contracts
+- New default middleware named MetaRefreshMiddleware that handles meta-refresh html tag redirections,
+- MetaRefreshMiddleware and RedirectMiddleware have different priorities to address #62
+- added from_crawler method to spiders
+- added system tests with mock server
+- more improvements to macOS compatibility (thanks Alex Cepoi)
+- several more cleanups to singletons and multi-spider support (thanks Nicolas Ramirez)
+- support custom download slots
+- added --spider option to "shell" command.
+- log overridden settings when Scrapy starts
+
+Thanks to everyone who contribute to this release. Here is a list of
+contributors sorted by number of commits::
+
+ 130 Pablo Hoffman <pablo@...>
+ 97 Daniel Graña <dangra@...>
+ 20 Nicolás Ramírez <nramirez.uy@...>
+ 13 Mikhail Korobov <kmike84@...>
+ 12 Pedro Faustino <pedrobandim@...>
+ 11 Steven Almeroth <sroth77@...>
+ 5 Rolando Espinoza La fuente <darkrho@...>
+ 4 Michal Danilak <mimino.coder@...>
+ 4 Alex Cepoi <alex.cepoi@...>
+ 4 Alexandr N Zamaraev (aka tonal) <tonal@...>
+ 3 paul <paul.tremberth@...>
+ 3 Martin Olveyra <molveyra@...>
+ 3 Jordi Llonch <llonchj@...>
+ 3 arijitchakraborty <myself.arijit@...>
+ 2 Shane Evans <shane.evans@...>
+ 2 joehillen <joehillen@...>
+ 2 Hart <HartSimha@...>
+ 2 Dan <ellisd23@...>
+ 1 Zuhao Wan <wanzuhao@...>
+ 1 whodatninja <blake@...>
+ 1 vkrest <v.krestiannykov@...>
+ 1 tpeng <pengtaoo@...>
+ 1 Tom Mortimer-Jones <tom@...>
+ 1 Rocio Aramberri <roschegel@...>
+ 1 Pedro <pedro@...>
+ 1 notsobad <wangxiaohugg@...>
+ 1 Natan L <kuyanatan.nlao@...>
+ 1 Mark Grey <mark.grey@...>
+ 1 Luan <luanpab@...>
+ 1 Libor Nenadál <libor.nenadal@...>
+ 1 Juan M Uys <opyate@...>
+ 1 Jonas Brunsgaard <jonas.brunsgaard@...>
+ 1 Ilya Baryshev <baryshev@...>
+ 1 Hasnain Lakhani <m.hasnain.lakhani@...>
+ 1 Emanuel Schorsch <emschorsch@...>
+ 1 Chris Tilden <chris.tilden@...>
+ 1 Capi Etheriel <barraponto@...>
+ 1 cacovsky <amarquesferraz@...>
+ 1 Berend Iwema <berend@...>
+
+
+Scrapy 0.16.5 (2013-05-30)
+--------------------------
+
+- obey request method when Scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)
+- fix inaccurate downloader middleware documentation. refs #280 (:commit:`40667cb`)
+- doc: remove links to diveintopython.org, which is no longer available. closes #246 (:commit:`bd58bfa`)
+- Find form nodes in invalid html5 documents (:commit:`e3d6945`)
+- Fix typo labeling attrs type bool instead of list (:commit:`a274276`)
+
+Scrapy 0.16.4 (2013-01-23)
+--------------------------
+
+- fixes spelling errors in documentation (:commit:`6d2b3aa`)
+- add doc about disabling an extension. refs #132 (:commit:`c90de33`)
+- Fixed error message formatting. log.err() doesn't support cool formatting and when error occurred, the message was: "ERROR: Error processing %(item)s" (:commit:`c16150c`)
+- lint and improve images pipeline error logging (:commit:`56b45fc`)
+- fixed doc typos (:commit:`243be84`)
+- add documentation topics: Broad Crawls & Common Practices (:commit:`1fbb715`)
+- fix bug in Scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)
+- Update docs/topics/commands.rst (:commit:`28eac7a`)
+
+Scrapy 0.16.3 (2012-12-07)
+--------------------------
+
+- Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced (:commit:`487b9b5`)
+- add error details when image pipeline fails (:commit:`8232569`)
+- improve macOS compatibility (:commit:`8dcf8aa`)
+- setup.py: use README.rst to populate long_description (:commit:`7b5310d`)
+- doc: removed obsolete references to ClientForm (:commit:`80f9bb6`)
+- correct docs for default storage backend (:commit:`2aa491b`)
+- doc: removed broken proxyhub link from FAQ (:commit:`bdf61c4`)
+- Fixed docs typo in SpiderOpenCloseLogging example (:commit:`7184094`)
+
+
+Scrapy 0.16.2 (2012-11-09)
+--------------------------
+
+- Scrapy contracts: python2.6 compat (:commit:`a4a9199`)
+- Scrapy contracts verbose option (:commit:`ec41673`)
+- proper unittest-like output for Scrapy contracts (:commit:`86635e4`)
+- added open_in_browser to debugging doc (:commit:`c9b690d`)
+- removed reference to global Scrapy stats from settings doc (:commit:`dd55067`)
+- Fix SpiderState bug in Windows platforms (:commit:`58998f4`)
+
+
+Scrapy 0.16.1 (2012-10-26)
+--------------------------
+
+- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release (:commit:`8c780fd`)
+- better backward compatibility for scrapy.conf.settings (:commit:`3403089`)
+- extended documentation on how to access crawler stats from extensions (:commit:`c4da0b5`)
+- removed .hgtags (no longer needed now that Scrapy uses git) (:commit:`d52c188`)
+- fix dashes under rst headers (:commit:`fa4f7f9`)
+- set release date for 0.16.0 in news (:commit:`e292246`)
+
+
+Scrapy 0.16.0 (2012-10-18)
+--------------------------
+
+Scrapy changes:
+
+- added :ref:`topics-contracts`, a mechanism for testing spiders in a formal/reproducible way
+- added options ``-o`` and ``-t`` to the :command:`runspider` command
+- documented :doc:`topics/autothrottle` and added to extensions installed by default. You still need to enable it with :setting:`AUTOTHROTTLE_ENABLED`
+- major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (``stats_spider_opened``, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.
+- added a ``process_start_requests()`` method to spider middlewares
+- dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.
+- dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.
+- documented :ref:`topics-api`
+- ``lxml`` is now the default selectors backend instead of ``libxml2``
+- ported FormRequest.from_response() to use `lxml`_ instead of `ClientForm`_
+- removed modules: ``scrapy.xlib.BeautifulSoup`` and ``scrapy.xlib.ClientForm``
+- SitemapSpider: added support for sitemap urls ending in .xml and .xml.gz, even if they advertise a wrong content type (:commit:`10ed28b`)
+- StackTraceDump extension: also dump trackref live references (:commit:`fe2ce93`)
+- nested items now fully supported in JSON and JSONLines exporters
+- added :reqmeta:`cookiejar` Request meta key to support multiple cookie sessions per spider
+- decoupled encoding detection code to `w3lib.encoding`_, and ported Scrapy code to use that module
+- dropped support for Python 2.5. See https://www.zyte.com/blog/scrapy-0-15-dropping-support-for-python-2-5/
+- dropped support for Twisted 2.5
+- added :setting:`REFERER_ENABLED` setting, to control referer middleware
+- changed default user agent to: ``Scrapy/VERSION (+http://scrapy.org)``
+- removed (undocumented) ``HTMLImageLinkExtractor`` class from ``scrapy.contrib.linkextractors.image``
+- removed per-spider settings (to be replaced by instantiating multiple crawler objects)
+- ``USER_AGENT`` spider attribute will no longer work, use ``user_agent`` attribute instead
+- ``DOWNLOAD_TIMEOUT`` spider attribute will no longer work, use ``download_timeout`` attribute instead
+- removed ``ENCODING_ALIASES`` setting, as encoding auto-detection has been moved to the `w3lib`_ library
+- promoted :ref:`topics-djangoitem` to main contrib
+- LogFormatter method now return dicts(instead of strings) to support lazy formatting (:gh:`164`, :commit:`dcef7b0`)
+- downloader handlers (:setting:`DOWNLOAD_HANDLERS` setting) now receive settings as the first argument of the ``__init__`` method
+- replaced memory usage accounting with (more portable) `resource`_ module, removed ``scrapy.utils.memory`` module
+- removed signal: ``scrapy.mail.mail_sent``
+- removed ``TRACK_REFS`` setting, now :ref:`trackrefs <topics-leaks-trackrefs>` is always enabled
+- DBM is now the default storage backend for HTTP cache middleware
+- number of log messages (per level) are now tracked through Scrapy stats (stat name: ``log_count/LEVEL``)
+- number received responses are now tracked through Scrapy stats (stat name: ``response_received_count``)
+- removed ``scrapy.log.started`` attribute
+
+Scrapy 0.14.4 (2012-05-15)
+--------------------------
+
+- added precise to supported Ubuntu distros (:commit:`b7e46df`)
+- fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py (:commit:`340fbdb`)
+- meta tag attributes for content-type http equiv can be in any order. #123 (:commit:`0cb68af`)
+- replace "import Image" by more standard "from PIL import Image". closes #88 (:commit:`4d17048`)
+- return trial status as bin/runtests.sh exit value. #118 (:commit:`b7b2e7f`)
+
+Scrapy 0.14.3 (2012-04-19)
+--------------------------
+
+- forgot to include pydispatch license. #118 (:commit:`fd85f9c`)
+- include egg files used by testsuite in source distribution. #118 (:commit:`c897793`)
+- update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (:commit:`2548dcc`)
+- added note to docs/topics/firebug.rst about google directory being shut down (:commit:`668e352`)
+- don't discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)
+- do not fail handling unicode xpaths in libxml2 backed selectors (:commit:`b830e95`)
+- fixed minor mistake in Request objects documentation (:commit:`bf3c9ee`)
+- fixed minor defect in link extractors documentation (:commit:`ba14f38`)
+- removed some obsolete remaining code related to sqlite support in Scrapy (:commit:`0665175`)
+
+Scrapy 0.14.2 (2012-03-11)
+--------------------------
+
+- move buffer pointing to start of file before computing checksum. refs #92 (:commit:`6a5bef2`)
+- Compute image checksum before persisting images. closes #92 (:commit:`9817df1`)
+- remove leaking references in cached failures (:commit:`673a120`)
+- fixed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) (:commit:`11133e9`)
+- fixed struct.error on http compression middleware. closes #87 (:commit:`1423140`)
+- ajax crawling wasn't expanding for unicode urls (:commit:`0de3fb4`)
+- Catch ``start_requests()`` iterator errors. refs #83 (:commit:`454a21d`)
+- Speed-up libxml2 XPathSelector (:commit:`2fbd662`)
+- updated versioning doc according to recent changes (:commit:`0a070f5`)
+- scrapyd: fixed documentation link (:commit:`2b4e4c3`)
+- extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`)
+
+Scrapy 0.14.1 (2012-01-02)
+--------------------------
+
+- extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`)
+- bumped version to 0.14.1 (:commit:`6cb9e1c`)
+- fixed reference to tutorial directory (:commit:`4b86bd6`)
+- doc: removed duplicated callback argument from Request.replace() (:commit:`1aeccdd`)
+- fixed formatting of scrapyd doc (:commit:`8bf19e6`)
+- Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension (:commit:`14a8e6e`)
+- added comment about why we disable ssl on boto images upload (:commit:`5223575`)
+- SSL handshaking hangs when doing too many parallel connections to S3 (:commit:`63d583d`)
+- change tutorial to follow changes on dmoz site (:commit:`bcb3198`)
+- Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 (:commit:`98f3f87`)
+- allow spider to set autothrottle max concurrency (:commit:`175a4b5`)
+
+Scrapy 0.14 (2011-12-31)
+------------------------
+
+New features and settings
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Support for AJAX crawlable urls
+- New persistent scheduler that stores requests on disk, allowing to suspend and resume crawls (:rev:`2737`)
+- added ``-o`` option to ``scrapy crawl``, a shortcut for dumping scraped items into a file (or standard output using ``-``)
+- Added support for passing custom settings to Scrapyd ``schedule.json`` api (:rev:`2779`, :rev:`2783`)
+- New ``ChunkedTransferMiddleware`` (enabled by default) to support `chunked transfer encoding`_ (:rev:`2769`)
+- Add boto 2.0 support for S3 downloader handler (:rev:`2763`)
+- Added `marshal`_ to formats supported by feed exports (:rev:`2744`)
+- In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)
+- Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)
+ - ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:
+ - :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``
+ - check the documentation for more details
+- Added builtin caching DNS resolver (:rev:`2728`)
+- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)
+- Moved spider queues to scrapyd: ``scrapy.spiderqueue`` -> ``scrapyd.spiderqueue`` (:rev:`2708`)
+- Moved sqlite utils to scrapyd: ``scrapy.utils.sqlite`` -> ``scrapyd.sqlite`` (:rev:`2781`)
+- Real support for returning iterators on ``start_requests()`` method. The iterator is now consumed during the crawl when the spider is getting idle (:rev:`2704`)
+- Added :setting:`REDIRECT_ENABLED` setting to quickly enable/disable the redirect middleware (:rev:`2697`)
+- Added :setting:`RETRY_ENABLED` setting to quickly enable/disable the retry middleware (:rev:`2694`)
+- Added ``CloseSpider`` exception to manually close spiders (:rev:`2691`)
+- Improved encoding detection by adding support for HTML5 meta charset declaration (:rev:`2690`)
+- Refactored close spider behavior to wait for all downloads to finish and be processed by spiders, before closing the spider (:rev:`2688`)
+- Added ``SitemapSpider`` (see documentation in Spiders page) (:rev:`2658`)
+- Added ``LogStats`` extension for periodically logging basic stats (like crawled pages and scraped items) (:rev:`2657`)
+- Make handling of gzipped responses more robust (#319, :rev:`2643`). Now Scrapy will try and decompress as much as possible from a gzipped response, instead of failing with an ``IOError``.
+- Simplified !MemoryDebugger extension to use stats for dumping memory debugging info (:rev:`2639`)
+- Added new command to edit spiders: ``scrapy edit`` (:rev:`2636`) and ``-e`` flag to ``genspider`` command that uses it (:rev:`2653`)
+- Changed default representation of items to pretty-printed dicts. (:rev:`2631`). This improves default logging by making log more readable in the default case, for both Scraped and Dropped lines.
+- Added :signal:`spider_error` signal (:rev:`2628`)
+- Added :setting:`COOKIES_ENABLED` setting (:rev:`2625`)
+- Stats are now dumped to Scrapy log (default value of :setting:`STATS_DUMP` setting has been changed to ``True``). This is to make Scrapy users more aware of Scrapy stats and the data that is collected there.
+- Added support for dynamically adjusting download delay and maximum concurrent requests (:rev:`2599`)
+- Added new DBM HTTP cache storage backend (:rev:`2576`)
+- Added ``listjobs.json`` API to Scrapyd (:rev:`2571`)
+- ``CsvItemExporter``: added ``join_multivalued`` parameter (:rev:`2578`)
+- Added namespace support to ``xmliter_lxml`` (:rev:`2552`)
+- Improved cookies middleware by making ``COOKIES_DEBUG`` nicer and documenting it (:rev:`2579`)
+- Several improvements to Scrapyd and Link extractors
+
+Code rearranged and removed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Merged item passed and item scraped concepts, as they have often proved confusing in the past. This means: (:rev:`2630`)
+ - original item_scraped signal was removed
+ - original item_passed signal was renamed to item_scraped
+ - old log lines ``Scraped Item...`` were removed
+ - old log lines ``Passed Item...`` were renamed to ``Scraped Item...`` lines and downgraded to ``DEBUG`` level
+- Reduced Scrapy codebase by striping part of Scrapy code into two new libraries:
+ - `w3lib`_ (several functions from ``scrapy.utils.{http,markup,multipart,response,url}``, done in :rev:`2584`)
+ - `scrapely`_ (was ``scrapy.contrib.ibl``, done in :rev:`2586`)
+- Removed unused function: ``scrapy.utils.request.request_info()`` (:rev:`2577`)
+- Removed googledir project from ``examples/googledir``. There's now a new example project called ``dirbot`` available on GitHub: https://github.com/scrapy/dirbot
+- Removed support for default field values in Scrapy items (:rev:`2616`)
+- Removed experimental crawlspider v2 (:rev:`2632`)
+- Removed scheduler middleware to simplify architecture. Duplicates filter is now done in the scheduler itself, using the same dupe filtering class as before (``DUPEFILTER_CLASS`` setting) (:rev:`2640`)
+- Removed support for passing urls to ``scrapy crawl`` command (use ``scrapy parse`` instead) (:rev:`2704`)
+- Removed deprecated Execution Queue (:rev:`2704`)
+- Removed (undocumented) spider context extension (from scrapy.contrib.spidercontext) (:rev:`2780`)
+- removed ``CONCURRENT_SPIDERS`` setting (use scrapyd maxproc instead) (:rev:`2789`)
+- Renamed attributes of core components: downloader.sites -> downloader.slots, scraper.sites -> scraper.slots (:rev:`2717`, :rev:`2718`)
+- Renamed setting ``CLOSESPIDER_ITEMPASSED`` to :setting:`CLOSESPIDER_ITEMCOUNT` (:rev:`2655`). Backward compatibility kept.
+
+Scrapy 0.12 (2011-01-02)
+------------------------
+
+The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
+
+New features and improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Passed item is now sent in the ``item`` argument of the :signal:`item_passed
+ <item_scraped>` (#273)
+- Added verbose option to ``scrapy version`` command, useful for bug reports (#298)
+- HTTP cache now stored by default in the project data dir (#279)
+- Added project data storage directory (#276, #277)
+- Documented file structure of Scrapy projects (see command-line tool doc)
+- New lxml backend for XPath selectors (#147)
+- Per-spider settings (#245)
+- Support exit codes to signal errors in Scrapy commands (#248)
+- Added ``-c`` argument to ``scrapy shell`` command
+- Made ``libxml2`` optional (#260)
+- New ``deploy`` command (#261)
+- Added :setting:`CLOSESPIDER_PAGECOUNT` setting (#253)
+- Added :setting:`CLOSESPIDER_ERRORCOUNT` setting (#254)
+
+Scrapyd changes
+~~~~~~~~~~~~~~~
+
+- Scrapyd now uses one process per spider
+- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default)
+- A minimal web ui was added, available at http://localhost:6800 by default
+- There is now a ``scrapy server`` command to start a Scrapyd server of the current project
+
+Changes to settings
+~~~~~~~~~~~~~~~~~~~
+
+- added ``HTTPCACHE_ENABLED`` setting (False by default) to enable HTTP cache middleware
+- changed ``HTTPCACHE_EXPIRATION_SECS`` semantics: now zero means "never expire".
+
+Deprecated/obsoleted functionality
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Deprecated ``runserver`` command in favor of ``server`` command which starts a Scrapyd server. See also: Scrapyd changes
+- Deprecated ``queue`` command in favor of using Scrapyd ``schedule.json`` API. See also: Scrapyd changes
+- Removed the !LxmlItemLoader (experimental contrib which never graduated to main contrib)
+
+Scrapy 0.10 (2010-09-11)
+------------------------
+
+The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
+
+New features and improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- New Scrapy service called ``scrapyd`` for deploying Scrapy crawlers in production (#218) (documentation available)
+- Simplified Images pipeline usage which doesn't require subclassing your own images pipeline now (#217)
+- Scrapy shell now shows the Scrapy log by default (#206)
+- Refactored execution queue in a common base code and pluggable backends called "spider queues" (#220)
+- New persistent spider queue (based on SQLite) (#198), available by default, which allows to start Scrapy in server mode and then schedule spiders to run.
+- Added documentation for Scrapy command-line tool and all its available sub-commands. (documentation available)
+- Feed exporters with pluggable backends (#197) (documentation available)
+- Deferred signals (#193)
+- Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195)
+- Support for overriding default request headers per spider (#181)
+- Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186)
+- Split Debian package into two packages - the library and the service (#187)
+- Scrapy log refactoring (#188)
+- New extension for keeping persistent spider contexts among different runs (#203)
+- Added ``dont_redirect`` request.meta key for avoiding redirects (#233)
+- Added ``dont_retry`` request.meta key for avoiding retries (#234)
+
+Command-line tool changes
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- New ``scrapy`` command which replaces the old ``scrapy-ctl.py`` (#199)
+ - there is only one global ``scrapy`` command now, instead of one ``scrapy-ctl.py`` per project
+ - Added ``scrapy.bat`` script for running more conveniently from Windows
+- Added bash completion to command-line tool (#210)
+- Renamed command ``start`` to ``runserver`` (#209)
+
+API changes
+~~~~~~~~~~~
+
+- ``url`` and ``body`` attributes of Request objects are now read-only (#230)
+- ``Request.copy()`` and ``Request.replace()`` now also copies their ``callback`` and ``errback`` attributes (#231)
+- Removed ``UrlFilterMiddleware`` from ``scrapy.contrib`` (already disabled by default)
+- Offsite middleware doesn't filter out any request coming from a spider that doesn't have a allowed_domains attribute (#225)
+- Removed Spider Manager ``load()`` method. Now spiders are loaded in the ``__init__`` method itself.
+- Changes to Scrapy Manager (now called "Crawler"):
+ - ``scrapy.core.manager.ScrapyManager`` class renamed to ``scrapy.crawler.Crawler``
+ - ``scrapy.core.manager.scrapymanager`` singleton moved to ``scrapy.project.crawler``
+- Moved module: ``scrapy.contrib.spidermanager`` to ``scrapy.spidermanager``
+- Spider Manager singleton moved from ``scrapy.spider.spiders`` to the ``spiders`` attribute of ``scrapy.project.crawler`` singleton.
+- moved Stats Collector classes: (#204)
+ - ``scrapy.stats.collector.StatsCollector`` to ``scrapy.statscol.StatsCollector``
+ - ``scrapy.stats.collector.SimpledbStatsCollector`` to ``scrapy.contrib.statscol.SimpledbStatsCollector``
+- default per-command settings are now specified in the ``default_settings`` attribute of command object class (#201)
+- changed arguments of Item pipeline ``process_item()`` method from ``(spider, item)`` to ``(item, spider)``
+ - backward compatibility kept (with deprecation warning)
+- moved ``scrapy.core.signals`` module to ``scrapy.signals``
+ - backward compatibility kept (with deprecation warning)
+- moved ``scrapy.core.exceptions`` module to ``scrapy.exceptions``
+ - backward compatibility kept (with deprecation warning)
+- added ``handles_request()`` class method to ``BaseSpider``
+- dropped ``scrapy.log.exc()`` function (use ``scrapy.log.err()`` instead)
+- dropped ``component`` argument of ``scrapy.log.msg()`` function
+- dropped ``scrapy.log.log_level`` attribute
+- Added ``from_settings()`` class methods to Spider Manager, and Item Pipeline Manager
+
+Changes to settings
+~~~~~~~~~~~~~~~~~~~
+
+- Added ``HTTPCACHE_IGNORE_SCHEMES`` setting to ignore certain schemes on !HttpCacheMiddleware (#225)
+- Added ``SPIDER_QUEUE_CLASS`` setting which defines the spider queue to use (#220)
+- Added ``KEEP_ALIVE`` setting (#220)
+- Removed ``SERVICE_QUEUE`` setting (#220)
+- Removed ``COMMANDS_SETTINGS_MODULE`` setting (#201)
+- Renamed ``REQUEST_HANDLERS`` to ``DOWNLOAD_HANDLERS`` and make download handlers classes (instead of functions)
+
+Scrapy 0.9 (2010-06-28)
+-----------------------
+
+The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
+
+New features and improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Added SMTP-AUTH support to scrapy.mail
+- New settings added: ``MAIL_USER``, ``MAIL_PASS`` (:rev:`2065` | #149)
+- Added new scrapy-ctl view command - To view URL in the browser, as seen by Scrapy (:rev:`2039`)
+- Added web service for controlling Scrapy process (this also deprecates the web console. (:rev:`2053` | #167)
+- Support for running Scrapy as a service, for production systems (:rev:`1988`, :rev:`2054`, :rev:`2055`, :rev:`2056`, :rev:`2057` | #168)
+- Added wrapper induction library (documentation only available in source code for now). (:rev:`2011`)
+- Simplified and improved response encoding support (:rev:`1961`, :rev:`1969`)
+- Added ``LOG_ENCODING`` setting (:rev:`1956`, documentation available)
+- Added ``RANDOMIZE_DOWNLOAD_DELAY`` setting (enabled by default) (:rev:`1923`, doc available)
+- ``MailSender`` is no longer IO-blocking (:rev:`1955` | #146)
+- Linkextractors and new Crawlspider now handle relative base tag urls (:rev:`1960` | #148)
+- Several improvements to Item Loaders and processors (:rev:`2022`, :rev:`2023`, :rev:`2024`, :rev:`2025`, :rev:`2026`, :rev:`2027`, :rev:`2028`, :rev:`2029`, :rev:`2030`)
+- Added support for adding variables to telnet console (:rev:`2047` | #165)
+- Support for requests without callbacks (:rev:`2050` | #166)
+
+API changes
+~~~~~~~~~~~
+
+- Change ``Spider.domain_name`` to ``Spider.name`` (SEP-012, :rev:`1975`)
+- ``Response.encoding`` is now the detected encoding (:rev:`1961`)
+- ``HttpErrorMiddleware`` now returns None or raises an exception (:rev:`2006` | #157)
+- ``scrapy.command`` modules relocation (:rev:`2035`, :rev:`2036`, :rev:`2037`)
+- Added ``ExecutionQueue`` for feeding spiders to scrape (:rev:`2034`)
+- Removed ``ExecutionEngine`` singleton (:rev:`2039`)
+- Ported ``S3ImagesStore`` (images pipeline) to use boto and threads (:rev:`2033`)
+- Moved module: ``scrapy.management.telnet`` to ``scrapy.telnet`` (:rev:`2047`)
+
+Changes to default settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Changed default ``SCHEDULER_ORDER`` to ``DFO`` (:rev:`1939`)
+
+Scrapy 0.8 (2009-12-12)
+-----------------------
+
+The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available.
+
+New features
+~~~~~~~~~~~~
+
+- Added DEFAULT_RESPONSE_ENCODING setting (:rev:`1809`)
+- Added ``dont_click`` argument to ``FormRequest.from_response()`` method (:rev:`1813`, :rev:`1816`)
+- Added ``clickdata`` argument to ``FormRequest.from_response()`` method (:rev:`1802`, :rev:`1803`)
+- Added support for HTTP proxies (``HttpProxyMiddleware``) (:rev:`1781`, :rev:`1785`)
+- Offsite spider middleware now logs messages when filtering out requests (:rev:`1841`)
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Changed ``scrapy.utils.response.get_meta_refresh()`` signature (:rev:`1804`)
+- Removed deprecated ``scrapy.item.ScrapedItem`` class - use ``scrapy.item.Item instead`` (:rev:`1838`)
+- Removed deprecated ``scrapy.xpath`` module - use ``scrapy.selector`` instead. (:rev:`1836`)
+- Removed deprecated ``core.signals.domain_open`` signal - use ``core.signals.domain_opened`` instead (:rev:`1822`)
+- ``log.msg()`` now receives a ``spider`` argument (:rev:`1822`)
+ - Old domain argument has been deprecated and will be removed in 0.9. For spiders, you should always use the ``spider`` argument and pass spider references. If you really want to pass a string, use the ``component`` argument instead.
+- Changed core signals ``domain_opened``, ``domain_closed``, ``domain_idle``
+- Changed Item pipeline to use spiders instead of domains
+ - The ``domain`` argument of ``process_item()`` item pipeline method was changed to ``spider``, the new signature is: ``process_item(spider, item)`` (:rev:`1827` | #105)
+ - To quickly port your code (to work with Scrapy 0.8) just use ``spider.domain_name`` where you previously used ``domain``.
+- Changed Stats API to use spiders instead of domains (:rev:`1849` | #113)
+ - ``StatsCollector`` was changed to receive spider references (instead of domains) in its methods (``set_value``, ``inc_value``, etc).
+ - added ``StatsCollector.iter_spider_stats()`` method
+ - removed ``StatsCollector.list_domains()`` method
+ - Also, Stats signals were renamed and now pass around spider references (instead of domains). Here's a summary of the changes:
+ - To quickly port your code (to work with Scrapy 0.8) just use ``spider.domain_name`` where you previously used ``domain``. ``spider_stats`` contains exactly the same data as ``domain_stats``.
+- ``CloseDomain`` extension moved to ``scrapy.contrib.closespider.CloseSpider`` (:rev:`1833`)
+ - Its settings were also renamed:
+ - ``CLOSEDOMAIN_TIMEOUT`` to ``CLOSESPIDER_TIMEOUT``
+ - ``CLOSEDOMAIN_ITEMCOUNT`` to ``CLOSESPIDER_ITEMCOUNT``
+- Removed deprecated ``SCRAPYSETTINGS_MODULE`` environment variable - use ``SCRAPY_SETTINGS_MODULE`` instead (:rev:`1840`)
+- Renamed setting: ``REQUESTS_PER_DOMAIN`` to ``CONCURRENT_REQUESTS_PER_SPIDER`` (:rev:`1830`, :rev:`1844`)
+- Renamed setting: ``CONCURRENT_DOMAINS`` to ``CONCURRENT_SPIDERS`` (:rev:`1830`)
+- Refactored HTTP Cache middleware
+- HTTP Cache middleware has been heavily refactored, retaining the same functionality except for the domain sectorization which was removed. (:rev:`1843` )
+- Renamed exception: ``DontCloseDomain`` to ``DontCloseSpider`` (:rev:`1859` | #120)
+- Renamed extension: ``DelayedCloseDomain`` to ``SpiderCloseDelay`` (:rev:`1861` | #121)
+- Removed obsolete ``scrapy.utils.markup.remove_escape_chars`` function - use ``scrapy.utils.markup.replace_escape_chars`` instead (:rev:`1865`)
+
+Scrapy 0.7 (2009-10-07)
+-----------------------
+
+First release of Scrapy.
+
+
+.. _chunked transfer encoding: https://en.wikipedia.org/wiki/Chunked_transfer_encoding
+.. _ClientForm: https://pypi.org/project/ClientForm/
+.. _lxml: https://lxml.de/
+.. _marshal: https://docs.python.org/2/library/marshal.html
+.. _queuelib: https://github.com/scrapy/queuelib
+.. _resource: https://docs.python.org/2/library/resource.html
+.. _scrapely: https://github.com/scrapy/scrapely
+.. _w3lib: https://github.com/scrapy/w3lib
+.. _w3lib.encoding: https://github.com/scrapy/w3lib/blob/master/w3lib/encoding.py
diff --git a/docs/news/1.x.rst b/docs/news/1.x.rst
new file mode 100644
index 000000000..598c2e40e
--- /dev/null
+++ b/docs/news/1.x.rst
@@ -0,0 +1,2333 @@
+.. _news-1.x:
+
+Scrapy 1.x release notes
+========================
+
+.. _release-1.8.4:
+
+Scrapy 1.8.4 (2024-02-14)
+-------------------------
+
+**Security bug fixes:**
+
+- Due to its `ReDoS vulnerabilities`_, ``scrapy.utils.iterators.xmliter`` is
+ now deprecated in favor of :func:`~scrapy.utils.iterators.xmliter_lxml`,
+ which :class:`~scrapy.spiders.XMLFeedSpider` now uses.
+
+ To minimize the impact of this change on existing code,
+ :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
+ the node namespace as a prefix in the node name, and big files with highly
+ nested trees when using libxml2 2.7+.
+
+ Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
+
+- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
+ to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
+ advisory`_ for more information.
+
+- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, use of the
+ ``scrapy.downloadermiddlewares.decompression`` module is discouraged and
+ will trigger a warning.
+
+- The ``Authorization`` header is now dropped on redirects to a different
+ domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
+ information.
+
+ .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
+
+
+.. _release-1.8.3:
+
+Scrapy 1.8.3 (2022-07-25)
+-------------------------
+
+**Security bug fix:**
+
+- When :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
+ processes a request with :reqmeta:`proxy` metadata, and that
+ :reqmeta:`proxy` metadata includes proxy credentials,
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` sets
+ the ``Proxy-Authorization`` header, but only if that header is not already
+ set.
+
+ There are third-party proxy-rotation downloader middlewares that set
+ different :reqmeta:`proxy` metadata every time they process a request.
+
+ Because of request retries and redirects, the same request can be processed
+ by downloader middlewares more than once, including both
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and
+ any third-party proxy-rotation downloader middleware.
+
+ These third-party proxy-rotation downloader middlewares could change the
+ :reqmeta:`proxy` metadata of a request to a new value, but fail to remove
+ the ``Proxy-Authorization`` header from the previous value of the
+ :reqmeta:`proxy` metadata, causing the credentials of one proxy to be sent
+ to a different proxy.
+
+ To prevent the unintended leaking of proxy credentials, the behavior of
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` is now
+ as follows when processing a request:
+
+ - If the request being processed defines :reqmeta:`proxy` metadata that
+ includes credentials, the ``Proxy-Authorization`` header is always
+ updated to feature those credentials.
+
+ - If the request being processed defines :reqmeta:`proxy` metadata
+ without credentials, the ``Proxy-Authorization`` header is removed
+ *unless* it was originally defined for the same proxy URL.
+
+ To remove proxy credentials while keeping the same proxy URL, remove
+ the ``Proxy-Authorization`` header.
+
+ - If the request has no :reqmeta:`proxy` metadata, or that metadata is a
+ falsy value (e.g. ``None``), the ``Proxy-Authorization`` header is
+ removed.
+
+ It is no longer possible to set a proxy URL through the
+ :reqmeta:`proxy` metadata but set the credentials through the
+ ``Proxy-Authorization`` header. Set proxy credentials through the
+ :reqmeta:`proxy` metadata instead.
+
+
+.. _release-1.8.2:
+
+Scrapy 1.8.2 (2022-03-01)
+-------------------------
+
+**Security bug fixes:**
+
+- When a :class:`~scrapy.Request` object with cookies defined gets a
+ redirect response causing a new :class:`~scrapy.Request` object to be
+ scheduled, the cookies defined in the original
+ :class:`~scrapy.Request` object are no longer copied into the new
+ :class:`~scrapy.Request` object.
+
+ If you manually set the ``Cookie`` header on a
+ :class:`~scrapy.Request` object and the domain name of the redirect
+ URL is not an exact match for the domain of the URL of the original
+ :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
+ from the new :class:`~scrapy.Request` object.
+
+ The old behavior could be exploited by an attacker to gain access to your
+ cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
+ information.
+
+ .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8
+
+ .. note:: It is still possible to enable the sharing of cookies between
+ different domains with a shared domain suffix (e.g.
+ ``example.com`` and any subdomain) by defining the shared domain
+ suffix (e.g. ``example.com``) as the cookie domain when defining
+ your cookies. See the documentation of the
+ :class:`~scrapy.Request` class for more information.
+
+- When the domain of a cookie, either received in the ``Set-Cookie`` header
+ of a response or defined in a :class:`~scrapy.Request` object, is set
+ to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
+ ignored unless the cookie domain is the same as the request domain.
+
+ The old behavior could be exploited by an attacker to inject cookies into
+ your requests to some other domains. Please, see the `mfjm-vh54-3f96
+ security advisory`_ for more information.
+
+ .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96
+
+
+.. _release-1.8.1:
+
+Scrapy 1.8.1 (2021-10-05)
+-------------------------
+
+* **Security bug fix:**
+
+ If you use
+ :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
+ (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP
+ authentication, any request exposes your credentials to the request target.
+
+ To prevent unintended exposure of authentication credentials to unintended
+ domains, you must now additionally set a new, additional spider attribute,
+ ``http_auth_domain``, and point it to the specific domain to which the
+ authentication credentials must be sent.
+
+ If the ``http_auth_domain`` spider attribute is not set, the domain of the
+ first request will be considered the HTTP authentication target, and
+ authentication credentials will only be sent in requests targeting that
+ domain.
+
+ If you need to send the same HTTP authentication credentials to multiple
+ domains, you can use :func:`w3lib.http.basic_auth_header` instead to
+ set the value of the ``Authorization`` header of your requests.
+
+ If you *really* want your spider to send the same HTTP authentication
+ credentials to any domain, set the ``http_auth_domain`` spider attribute
+ to ``None``.
+
+ Finally, if you are a user of `scrapy-splash`_, know that this version of
+ Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will
+ need to upgrade scrapy-splash to a greater version for it to continue to
+ work.
+
+.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
+
+
+.. _release-1.8.0:
+
+Scrapy 1.8.0 (2019-10-28)
+-------------------------
+
+Highlights:
+
+* Dropped Python 3.4 support and updated minimum requirements; made Python 3.8
+ support official
+* New :meth:`.Request.from_curl` class method
+* New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings
+* New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
+ :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. skip: start
+
+* Python 3.4 is no longer supported, and some of the minimum requirements of
+ Scrapy have also changed:
+
+ * :doc:`cssselect <cssselect:index>` 0.9.1
+ * cryptography_ 2.0
+ * lxml_ 3.5.0
+ * pyOpenSSL_ 16.2.0
+ * queuelib_ 1.4.2
+ * service_identity_ 16.0.0
+ * six_ 1.10.0
+ * Twisted_ 17.9.0 (16.0.0 with Python 2)
+ * zope.interface_ 4.1.3
+
+ (:gh:`3892`)
+
+* ``JSONRequest`` is now called :class:`~scrapy.http.JsonRequest` for
+ consistency with similar classes (:gh:`3929`, :gh:`3982`)
+
+* If you are using a custom context factory
+ (``DOWNLOADER_CLIENTCONTEXTFACTORY``), its ``__init__`` method must
+ accept two new parameters: ``tls_verbose_logging`` and ``tls_ciphers``
+ (:gh:`2111`, :gh:`3392`, :gh:`3442`, :gh:`3450`)
+
+* :class:`~scrapy.loader.ItemLoader` now turns the values of its input item
+ into lists:
+
+ .. code-block:: pycon
+
+ >>> item = MyItem()
+ >>> item["field"] = "value1"
+ >>> loader = ItemLoader(item=item)
+ >>> item["field"]
+ ['value1']
+
+ This is needed to allow adding values to existing fields
+ (``loader.add_value('field', 'value2')``).
+
+ (:gh:`3804`, :gh:`3819`, :gh:`3897`, :gh:`3976`, :gh:`3998`,
+ :gh:`4036`)
+
+.. skip: end
+
+See also :ref:`1.8-deprecation-removals` below.
+
+
+New features
+~~~~~~~~~~~~
+
+* A new :meth:`Request.from_curl <scrapy.Request.from_curl>` class
+ method allows :ref:`creating a request from a cURL command
+ <requests-from-curl>` (:gh:`2985`, :gh:`3862`)
+
+* A new :setting:`ROBOTSTXT_PARSER` setting allows choosing which robots.txt_
+ parser to use. It includes built-in support for
+ :ref:`RobotFileParser <python-robotfileparser>`,
+ :ref:`Protego <protego-parser>` (default), Reppy, and
+ :ref:`Robotexclusionrulesparser <rerp-parser>`, and allows you to
+ :ref:`implement support for additional parsers
+ <support-for-new-robots-parser>` (:gh:`754`, :gh:`2669`,
+ :gh:`3796`, :gh:`3935`, :gh:`3969`, :gh:`4006`)
+
+* A new :setting:`ROBOTSTXT_USER_AGENT` setting allows defining a separate
+ user agent string to use for robots.txt_ parsing (:gh:`3931`,
+ :gh:`3966`)
+
+* :class:`~scrapy.spiders.Rule` no longer requires a :class:`LinkExtractor
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` parameter
+ (:gh:`781`, :gh:`4016`)
+
+* Use the new :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting to customize
+ the TLS/SSL ciphers used by the default HTTP/1.1 downloader (:gh:`3392`,
+ :gh:`3442`)
+
+* Set the new :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` setting to
+ ``True`` to enable debug-level messages about TLS connection parameters
+ after establishing HTTPS connections (:gh:`2111`, :gh:`3450`)
+
+* Callbacks that receive keyword arguments (see :attr:`.Request.cb_kwargs`)
+ can now be tested using the new :class:`@cb_kwargs
+ <scrapy.contracts.default.CallbackKeywordArgumentsContract>`
+ :ref:`spider contract <topics-contracts>` (:gh:`3985`, :gh:`3988`)
+
+* When a :class:`@scrapes <scrapy.contracts.default.ScrapesContract>` spider
+ contract fails, all missing fields are now reported (:gh:`766`,
+ :gh:`3939`)
+
+* :ref:`Custom log formats <custom-log-formats>` can now drop messages by
+ having the corresponding methods of the configured :setting:`LOG_FORMATTER`
+ return ``None`` (:gh:`3984`, :gh:`3987`)
+
+* A much improved completion definition is now available for Zsh_
+ (:gh:`4069`)
+
+
+Bug fixes
+~~~~~~~~~
+
+* :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` no
+ longer makes later calls to :meth:`ItemLoader.get_output_value()
+ <scrapy.loader.ItemLoader.get_output_value>` or
+ :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` return
+ empty data (:gh:`3804`, :gh:`3819`, :gh:`3897`, :gh:`3976`,
+ :gh:`3998`, :gh:`4036`)
+
+* Fixed :class:`~scrapy.statscollectors.DummyStatsCollector` raising a
+ :exc:`TypeError` exception (:gh:`4007`, :gh:`4052`)
+
+* :meth:`FilesPipeline.file_path
+ <scrapy.pipelines.files.FilesPipeline.file_path>` and
+ :meth:`ImagesPipeline.file_path
+ <scrapy.pipelines.images.ImagesPipeline.file_path>` no longer choose
+ file extensions that are not `registered with IANA`_ (:gh:`1287`,
+ :gh:`3953`, :gh:`3954`)
+
+* When using botocore_ to persist files in S3, all botocore-supported headers
+ are properly mapped now (:gh:`3904`, :gh:`3905`)
+
+* FTP passwords in :setting:`FEED_URI` containing percent-escaped characters
+ are now properly decoded (:gh:`3941`)
+
+* A memory-handling and error-handling issue in
+ ``scrapy.utils.ssl.get_temp_key_info()`` has been fixed (:gh:`3920`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* The documentation now covers how to define and configure a :ref:`custom log
+ format <custom-log-formats>` (:gh:`3616`, :gh:`3660`)
+
+* API documentation added for :class:`~scrapy.exporters.MarshalItemExporter`
+ and :class:`~scrapy.exporters.PythonItemExporter` (:gh:`3973`)
+
+* API documentation added for :class:`~scrapy.item.BaseItem` and
+ :class:`~scrapy.item.ItemMeta` (:gh:`3999`)
+
+* Minor documentation fixes (:gh:`2998`, :gh:`3398`, :gh:`3597`,
+ :gh:`3894`, :gh:`3934`, :gh:`3978`, :gh:`3993`, :gh:`4022`,
+ :gh:`4028`, :gh:`4033`, :gh:`4046`, :gh:`4050`, :gh:`4055`,
+ :gh:`4056`, :gh:`4061`, :gh:`4072`, :gh:`4071`, :gh:`4079`,
+ :gh:`4081`, :gh:`4089`, :gh:`4093`)
+
+
+.. _1.8-deprecation-removals:
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* ``scrapy.xlib`` has been removed (:gh:`4015`)
+
+
+.. _1.8-deprecations:
+
+Deprecations
+~~~~~~~~~~~~
+
+* The LevelDB_ storage backend
+ (``scrapy.extensions.httpcache.LeveldbCacheStorage``) of
+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` is
+ deprecated (:gh:`4085`, :gh:`4092`)
+
+* Use of the undocumented ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
+ variable is deprecated (:gh:`3910`)
+
+* ``scrapy.item.DictItem`` is deprecated, use :class:`~scrapy.item.Item`
+ instead (:gh:`3999`)
+
+
+Other changes
+~~~~~~~~~~~~~
+
+* Minimum versions of optional Scrapy requirements that are covered by
+ continuous integration tests have been updated:
+
+ * botocore_ 1.3.23
+ * Pillow_ 3.4.2
+
+ Lower versions of these optional requirements may work, but it is not
+ guaranteed (:gh:`3892`)
+
+* GitHub templates for bug reports and feature requests (:gh:`3126`,
+ :gh:`3471`, :gh:`3749`, :gh:`3754`)
+
+* Continuous integration fixes (:gh:`3923`)
+
+* Code cleanup (:gh:`3391`, :gh:`3907`, :gh:`3946`, :gh:`3950`,
+ :gh:`4023`, :gh:`4031`)
+
+
+.. _release-1.7.4:
+
+Scrapy 1.7.4 (2019-10-21)
+-------------------------
+
+Revert the fix for :gh:`3804` (:gh:`3819`), which has a few undesired
+side effects (:gh:`3897`, :gh:`3976`).
+
+As a result, when an item loader is initialized with an item,
+:meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` once again
+makes later calls to :meth:`ItemLoader.get_output_value()
+<scrapy.loader.ItemLoader.get_output_value>` or :meth:`ItemLoader.load_item()
+<scrapy.loader.ItemLoader.load_item>` return empty data.
+
+
+.. _release-1.7.3:
+
+Scrapy 1.7.3 (2019-08-01)
+-------------------------
+
+Enforce lxml 4.3.5 or lower for Python 3.4 (:gh:`3912`, :gh:`3918`).
+
+
+.. _release-1.7.2:
+
+Scrapy 1.7.2 (2019-07-23)
+-------------------------
+
+Fix Python 2 support (:gh:`3889`, :gh:`3893`, :gh:`3896`).
+
+
+.. _release-1.7.1:
+
+Scrapy 1.7.1 (2019-07-18)
+-------------------------
+
+Re-packaging of Scrapy 1.7.0, which was missing some changes in PyPI.
+
+
+.. _release-1.7.0:
+
+Scrapy 1.7.0 (2019-07-18)
+-------------------------
+
+.. note:: Make sure you install Scrapy 1.7.1. The Scrapy 1.7.0 package in PyPI
+ is the result of an erroneous commit tagging and does not include all
+ the changes described below.
+
+Highlights:
+
+* Improvements for crawls targeting multiple domains
+* A cleaner way to pass arguments to callbacks
+* A new class for JSON requests
+* Improvements for rule-based spiders
+* New features for feed exports
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* ``429`` is now part of the :setting:`RETRY_HTTP_CODES` setting by default
+
+ This change is **backward incompatible**. If you don’t want to retry
+ ``429``, you must override :setting:`RETRY_HTTP_CODES` accordingly.
+
+* :class:`~scrapy.crawler.Crawler`,
+ :meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
+ :meth:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
+ no longer accept a :class:`~scrapy.spiders.Spider` subclass instance, they
+ only accept a :class:`~scrapy.spiders.Spider` subclass now.
+
+ :class:`~scrapy.spiders.Spider` subclass instances were never meant to
+ work, and they were not working as one would expect: instead of using the
+ passed :class:`~scrapy.spiders.Spider` subclass instance, their
+ :class:`~scrapy.spiders.Spider.from_crawler` method was called to generate
+ a new instance.
+
+* Non-default values for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting
+ may stop working. Scheduler priority queue classes now need to handle
+ :class:`~scrapy.Request` objects instead of arbitrary Python data
+ structures.
+
+* An additional ``crawler`` parameter has been added to the ``__init__``
+ method of the :class:`~scrapy.core.scheduler.Scheduler` class. Custom
+ scheduler subclasses which don't accept arbitrary parameters in their
+ ``__init__`` method might break because of this change.
+
+ For more information, see :setting:`SCHEDULER`.
+
+See also :ref:`1.7-deprecation-removals` below.
+
+
+New features
+~~~~~~~~~~~~
+
+* A new scheduler priority queue,
+ ``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
+ :ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
+ scheduling improvement on crawls targeting multiple web domains, at the
+ cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:gh:`3520`)
+
+* A new :attr:`.Request.cb_kwargs` attribute
+ provides a cleaner way to pass keyword arguments to callback methods
+ (:gh:`1138`, :gh:`3563`)
+
+* A new :class:`JSONRequest <scrapy.http.JsonRequest>` class offers a more
+ convenient way to build JSON requests (:gh:`3504`, :gh:`3505`)
+
+* A ``process_request`` callback passed to the :class:`~scrapy.spiders.Rule`
+ ``__init__`` method now receives the :class:`~scrapy.http.Response` object that
+ originated the request as its second argument (:gh:`3682`)
+
+* A new ``restrict_text`` parameter for the
+ :attr:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ ``__init__`` method allows filtering links by linking text (:gh:`3622`,
+ :gh:`3635`)
+
+* A new :setting:`FEED_STORAGE_S3_ACL` setting allows defining a custom ACL
+ for feeds exported to Amazon S3 (:gh:`3607`)
+
+* A new :setting:`FEED_STORAGE_FTP_ACTIVE` setting allows using FTP’s active
+ connection mode for feeds exported to FTP servers (:gh:`3829`)
+
+* A new :setting:`METAREFRESH_IGNORE_TAGS` setting allows overriding which
+ HTML tags are ignored when searching a response for HTML meta tags that
+ trigger a redirect (:gh:`1422`, :gh:`3768`)
+
+* A new :reqmeta:`redirect_reasons` request meta key exposes the reason
+ (status code, meta refresh) behind every followed redirect (:gh:`3581`,
+ :gh:`3687`)
+
+* The ``SCRAPY_CHECK`` variable is now set to the ``true`` string during runs
+ of the :command:`check` command, which allows :ref:`detecting contract
+ check runs from code <detecting-contract-check-runs>` (:gh:`3704`,
+ :gh:`3739`)
+
+* A new :meth:`Item.deepcopy() <scrapy.item.Item.deepcopy>` method makes it
+ easier to :ref:`deep-copy items <copying-items>` (:gh:`1493`,
+ :gh:`3671`)
+
+* :class:`~scrapy.extensions.corestats.CoreStats` also logs
+ ``elapsed_time_seconds`` now (:gh:`3638`)
+
+* Exceptions from :class:`~scrapy.loader.ItemLoader` :ref:`input and output
+ processors <topics-loaders-processors>` are now more verbose
+ (:gh:`3836`, :gh:`3840`)
+
+* :class:`~scrapy.crawler.Crawler`,
+ :class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>` and
+ :class:`CrawlerRunner.create_crawler <scrapy.crawler.CrawlerRunner.create_crawler>`
+ now fail gracefully if they receive a :class:`~scrapy.spiders.Spider`
+ subclass instance instead of the subclass itself (:gh:`2283`,
+ :gh:`3610`, :gh:`3872`)
+
+
+Bug fixes
+~~~~~~~~~
+
+* :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
+ is now also invoked for generators (:gh:`220`, :gh:`2061`)
+
+* System exceptions like KeyboardInterrupt_ are no longer caught
+ (:gh:`3726`)
+
+* :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` no
+ longer makes later calls to :meth:`ItemLoader.get_output_value()
+ <scrapy.loader.ItemLoader.get_output_value>` or
+ :meth:`ItemLoader.load_item() <scrapy.loader.ItemLoader.load_item>` return
+ empty data (:gh:`3804`, :gh:`3819`)
+
+* The images pipeline (:class:`~scrapy.pipelines.images.ImagesPipeline`) no
+ longer ignores these Amazon S3 settings: :setting:`AWS_ENDPOINT_URL`,
+ :setting:`AWS_REGION_NAME`, :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`
+ (:gh:`3625`)
+
+* Fixed a memory leak in ``scrapy.pipelines.media.MediaPipeline`` affecting,
+ for example, non-200 responses and exceptions from custom middlewares
+ (:gh:`3813`)
+
+* Requests with private callbacks are now correctly unserialized from disk
+ (:gh:`3790`)
+
+* :meth:`.FormRequest.from_response`
+ now handles invalid methods like major web browsers (:gh:`3777`,
+ :gh:`3794`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* A new topic, :ref:`topics-dynamic-content`, covers recommended approaches
+ to read dynamically-loaded data (:gh:`3703`)
+
+* :ref:`topics-broad-crawls` now features information about memory usage
+ (:gh:`1264`, :gh:`3866`)
+
+* The documentation of :class:`~scrapy.spiders.Rule` now covers how to access
+ the text of a link when using :class:`~scrapy.spiders.CrawlSpider`
+ (:gh:`3711`, :gh:`3712`)
+
+* A new section, :ref:`httpcache-storage-custom`, covers writing a custom
+ cache storage backend for
+ :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`
+ (:gh:`3683`, :gh:`3692`)
+
+* A new :ref:`FAQ <faq>` entry, :ref:`faq-split-item`, explains what to do
+ when you want to split an item into multiple items from an item pipeline
+ (:gh:`2240`, :gh:`3672`)
+
+* Updated the :ref:`FAQ entry about crawl order <faq-bfo-dfo>` to explain why
+ the first few requests rarely follow the desired order (:gh:`1739`,
+ :gh:`3621`)
+
+* The :setting:`LOGSTATS_INTERVAL` setting (:gh:`3730`), the
+ :meth:`FilesPipeline.file_path <scrapy.pipelines.files.FilesPipeline.file_path>`
+ and
+ :meth:`ImagesPipeline.file_path <scrapy.pipelines.images.ImagesPipeline.file_path>`
+ methods (:gh:`2253`, :gh:`3609`) and the
+ :meth:`Crawler.stop() <scrapy.crawler.Crawler.stop>` method (:gh:`3842`)
+ are now documented
+
+* Some parts of the documentation that were confusing or misleading are now
+ clearer (:gh:`1347`, :gh:`1789`, :gh:`2289`, :gh:`3069`,
+ :gh:`3615`, :gh:`3626`, :gh:`3668`, :gh:`3670`, :gh:`3673`,
+ :gh:`3728`, :gh:`3762`, :gh:`3861`, :gh:`3882`)
+
+* Minor documentation fixes (:gh:`3648`, :gh:`3649`, :gh:`3662`,
+ :gh:`3674`, :gh:`3676`, :gh:`3694`, :gh:`3724`, :gh:`3764`,
+ :gh:`3767`, :gh:`3791`, :gh:`3797`, :gh:`3806`, :gh:`3812`)
+
+.. _1.7-deprecation-removals:
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+The following deprecated APIs have been removed (:gh:`3578`):
+
+* ``scrapy.conf`` (use :attr:`Crawler.settings
+ <scrapy.crawler.Crawler.settings>`)
+
+* From ``scrapy.core.downloader.handlers``:
+
+ * ``http.HttpDownloadHandler`` (use ``http10.HTTP10DownloadHandler``)
+
+* ``scrapy.loader.ItemLoader._get_values`` (use ``_get_xpathvalues``)
+
+* ``scrapy.loader.XPathItemLoader`` (use :class:`~scrapy.loader.ItemLoader`)
+
+* ``scrapy.log`` (see :ref:`topics-logging`)
+
+* From ``scrapy.pipelines``:
+
+ * ``files.FilesPipeline.file_key`` (use ``file_path``)
+
+ * ``images.ImagesPipeline.file_key`` (use ``file_path``)
+
+ * ``images.ImagesPipeline.image_key`` (use ``file_path``)
+
+ * ``images.ImagesPipeline.thumb_key`` (use ``thumb_path``)
+
+* From both ``scrapy.selector`` and ``scrapy.selector.lxmlsel``:
+
+ * ``HtmlXPathSelector`` (use :class:`~scrapy.Selector`)
+
+ * ``XmlXPathSelector`` (use :class:`~scrapy.Selector`)
+
+ * ``XPathSelector`` (use :class:`~scrapy.Selector`)
+
+ * ``XPathSelectorList`` (use :class:`~scrapy.Selector`)
+
+* From ``scrapy.selector.csstranslator``:
+
+ * ``ScrapyGenericTranslator`` (use parsel.csstranslator.GenericTranslator_)
+
+ * ``ScrapyHTMLTranslator`` (use parsel.csstranslator.HTMLTranslator_)
+
+ * ``ScrapyXPathExpr`` (use parsel.csstranslator.XPathExpr_)
+
+* From :class:`~scrapy.Selector`:
+
+ * ``_root`` (both the ``__init__`` method argument and the object property, use
+ ``root``)
+
+ * ``extract_unquoted`` (use ``getall``)
+
+ * ``select`` (use ``xpath``)
+
+* From :class:`~scrapy.selector.SelectorList`:
+
+ * ``extract_unquoted`` (use ``getall``)
+
+ * ``select`` (use ``xpath``)
+
+ * ``x`` (use ``xpath``)
+
+* ``scrapy.spiders.BaseSpider`` (use :class:`~scrapy.spiders.Spider`)
+
+* From :class:`~scrapy.spiders.Spider` (and subclasses):
+
+ * ``DOWNLOAD_DELAY`` (use :ref:`download_delay
+ <spider-download_delay-attribute>`)
+
+ * ``set_crawler`` (use :meth:`~scrapy.spiders.Spider.from_crawler`)
+
+* ``scrapy.spiders.spiders`` (use :class:`~scrapy.spiderloader.SpiderLoader`)
+
+* ``scrapy.telnet`` (use :mod:`scrapy.extensions.telnet`)
+
+* From ``scrapy.utils.python``:
+
+ * ``str_to_unicode`` (use ``to_unicode``)
+
+ * ``unicode_to_str`` (use ``to_bytes``)
+
+* ``scrapy.utils.response.body_or_str``
+
+The following deprecated settings have also been removed (:gh:`3578`):
+
+* ``SPIDER_MANAGER_CLASS`` (use :setting:`SPIDER_LOADER_CLASS`)
+
+
+.. _1.7-deprecations:
+
+Deprecations
+~~~~~~~~~~~~
+
+* The ``queuelib.PriorityQueue`` value for the
+ :setting:`SCHEDULER_PRIORITY_QUEUE` setting is deprecated. Use
+ ``scrapy.pqueues.ScrapyPriorityQueue`` instead.
+
+* ``process_request`` callbacks passed to :class:`~scrapy.spiders.Rule` that
+ do not accept two arguments are deprecated.
+
+* The following modules are deprecated:
+
+ * ``scrapy.utils.http`` (use `w3lib.http`_)
+
+ * ``scrapy.utils.markup`` (use `w3lib.html`_)
+
+ * ``scrapy.utils.multipart`` (use `urllib3`_)
+
+* The ``scrapy.utils.datatypes.MergeDict`` class is deprecated for Python 3
+ code bases. Use :class:`~collections.ChainMap` instead. (:gh:`3878`)
+
+* The ``scrapy.utils.gz.is_gzipped`` function is deprecated. Use
+ ``scrapy.utils.gz.gzip_magic_number`` instead.
+
+.. _urllib3: https://urllib3.readthedocs.io/en/latest/index.html
+.. _w3lib.html: https://w3lib.readthedocs.io/en/latest/w3lib.html#module-w3lib.html
+.. _w3lib.http: https://w3lib.readthedocs.io/en/latest/w3lib.html#module-w3lib.http
+
+
+Other changes
+~~~~~~~~~~~~~
+
+* It is now possible to run all tests from the same tox_ environment in
+ parallel; the documentation now covers :ref:`this and other ways to run
+ tests <running-tests>` (:gh:`3707`)
+
+* It is now possible to generate an API documentation coverage report
+ (:gh:`3806`, :gh:`3810`, :gh:`3860`)
+
+* The :ref:`documentation policies <documentation-policies>` now require
+ docstrings_ (:gh:`3701`) that follow `PEP 257`_ (:gh:`3748`)
+
+* Internal fixes and cleanup (:gh:`3629`, :gh:`3643`, :gh:`3684`,
+ :gh:`3698`, :gh:`3734`, :gh:`3735`, :gh:`3736`, :gh:`3737`,
+ :gh:`3809`, :gh:`3821`, :gh:`3825`, :gh:`3827`, :gh:`3833`,
+ :gh:`3857`, :gh:`3877`)
+
+.. _release-1.6.0:
+
+Scrapy 1.6.0 (2019-01-30)
+-------------------------
+
+Highlights:
+
+* better Windows support;
+* Python 3.7 compatibility;
+* big documentation improvements, including a switch
+ from ``.extract_first()`` + ``.extract()`` API to ``.get()`` + ``.getall()``
+ API;
+* feed exports, FilePipeline and MediaPipeline improvements;
+* better extensibility: :signal:`item_error` and
+ :signal:`request_reached_downloader` signals; ``from_crawler`` support
+ for feed exporters, feed storages and dupefilters.
+* ``scrapy.contracts`` fixes and new features;
+* telnet console security improvements, first released as a
+ backport in :ref:`release-1.5.2`;
+* clean-up of the deprecated code;
+* various bug fixes, small new features and usability improvements across
+ the codebase.
+
+Selector API changes
+~~~~~~~~~~~~~~~~~~~~
+
+While these are not changes in Scrapy itself, but rather in the parsel_
+library which Scrapy uses for xpath/css selectors, these changes are
+worth mentioning here. Scrapy now depends on parsel >= 1.5, and
+Scrapy documentation is updated to follow recent ``parsel`` API conventions.
+
+Most visible change is that ``.get()`` and ``.getall()`` selector
+methods are now preferred over ``.extract_first()`` and ``.extract()``.
+We feel that these new methods result in a more concise and readable code.
+See :ref:`old-extraction-api` for more details.
+
+.. note::
+ There are currently **no plans** to deprecate ``.extract()``
+ and ``.extract_first()`` methods.
+
+Another useful new feature is the introduction of ``Selector.attrib`` and
+``SelectorList.attrib`` properties, which make it easier to get
+attributes of HTML elements. See :ref:`selecting-attributes`.
+
+CSS selectors are cached in parsel >= 1.5, which makes them faster
+when the same CSS path is used many times. This is very common in
+case of Scrapy spiders: callbacks are usually called several times,
+on different pages.
+
+If you're using custom ``Selector`` or ``SelectorList`` subclasses,
+a **backward incompatible** change in parsel may affect your code.
+See `parsel changelog`_ for a detailed description, as well as for the
+full list of improvements.
+
+.. _parsel changelog: https://parsel.readthedocs.io/en/latest/history.html
+
+Telnet console
+~~~~~~~~~~~~~~
+
+**Backward incompatible**: Scrapy's telnet console now requires username
+and password. See :ref:`topics-telnetconsole` for more details. This change
+fixes a **security issue**; see :ref:`release-1.5.2` release notes for details.
+
+New extensibility features
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* ``from_crawler`` support is added to feed exporters and feed storages. This,
+ among other things, allows to access Scrapy settings from custom feed
+ storages and exporters (:gh:`1605`, :gh:`3348`).
+* ``from_crawler`` support is added to dupefilters (:gh:`2956`); this allows
+ to access e.g. settings or a spider from a dupefilter.
+* :signal:`item_error` is fired when an error happens in a pipeline
+ (:gh:`3256`);
+* :signal:`request_reached_downloader` is fired when Downloader gets
+ a new Request; this signal can be useful e.g. for custom Schedulers
+ (:gh:`3393`).
+* new SitemapSpider :meth:`~.SitemapSpider.sitemap_filter` method which allows
+ to select sitemap entries based on their attributes in SitemapSpider
+ subclasses (:gh:`3512`).
+* Lazy loading of Downloader Handlers is now optional; this enables better
+ initialization error handling in custom Downloader Handlers (:gh:`3394`).
+
+New FilePipeline and MediaPipeline features
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`,
+ :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`.
+ For example, this allows to use alternative or self-hosted
+ AWS-compatible providers (:gh:`2609`, :gh:`3548`).
+* ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and
+ :setting:`IMAGES_STORE_GCS_ACL` (:gh:`3199`).
+
+``scrapy.contracts`` improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Exceptions in contracts code are handled better (:gh:`3377`);
+* ``dont_filter=True`` is used for contract requests, which allows to test
+ different callbacks with the same URL (:gh:`3381`);
+* ``request_cls`` attribute in Contract subclasses allow to use different
+ Request classes in contracts, for example FormRequest (:gh:`3383`).
+* Fixed errback handling in contracts, e.g. for cases where a contract
+ is executed for URL which returns non-200 response (:gh:`3371`).
+
+Usability improvements
+~~~~~~~~~~~~~~~~~~~~~~
+
+* more stats for RobotsTxtMiddleware (:gh:`3100`)
+* INFO log level is used to show telnet host/port (:gh:`3115`)
+* a message is added to IgnoreRequest in RobotsTxtMiddleware (:gh:`3113`)
+* better validation of ``url`` argument in ``Response.follow`` (:gh:`3131`)
+* non-zero exit code is returned from Scrapy commands when error happens
+ on spider initialization (:gh:`3226`)
+* Link extraction improvements: "ftp" is added to scheme list (:gh:`3152`);
+ "flv" is added to common video extensions (:gh:`3165`)
+* better error message when an exporter is disabled (:gh:`3358`);
+* ``scrapy shell --help`` mentions syntax required for local files
+ (``./file.html``) - :gh:`3496`.
+* Referer header value is added to RFPDupeFilter log messages (:gh:`3588`)
+
+Bug fixes
+~~~~~~~~~
+
+* fixed issue with extra blank lines in .csv exports under Windows
+ (:gh:`3039`);
+* proper handling of pickling errors in Python 3 when serializing objects
+ for disk queues (:gh:`3082`)
+* flags are now preserved when copying Requests (:gh:`3342`);
+* FormRequest.from_response clickdata shouldn't ignore elements with
+ ``input[type=image]`` (:gh:`3153`).
+* FormRequest.from_response should preserve duplicate keys (:gh:`3247`)
+
+Documentation improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Docs are re-written to suggest .get/.getall API instead of
+ .extract/.extract_first. Also, :ref:`topics-selectors` docs are updated
+ and re-structured to match latest parsel docs; they now contain more topics,
+ such as :ref:`selecting-attributes` or :ref:`topics-selectors-css-extensions`
+ (:gh:`3390`).
+* :ref:`topics-developer-tools` is a new tutorial which replaces
+ old Firefox and Firebug tutorials (:gh:`3400`).
+* SCRAPY_PROJECT environment variable is documented (:gh:`3518`);
+* troubleshooting section is added to install instructions (:gh:`3517`);
+* improved links to beginner resources in the tutorial
+ (:gh:`3367`, :gh:`3468`);
+* fixed :setting:`RETRY_HTTP_CODES` default values in docs (:gh:`3335`);
+* remove unused ``DEPTH_STATS`` option from docs (:gh:`3245`);
+* other cleanups (:gh:`3347`, :gh:`3350`, :gh:`3445`, :gh:`3544`,
+ :gh:`3605`).
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+Compatibility shims for pre-1.0 Scrapy module names are removed
+(:gh:`3318`):
+
+* ``scrapy.command``
+* ``scrapy.contrib`` (with all submodules)
+* ``scrapy.contrib_exp`` (with all submodules)
+* ``scrapy.dupefilter``
+* ``scrapy.linkextractor``
+* ``scrapy.project``
+* ``scrapy.spider``
+* ``scrapy.spidermanager``
+* ``scrapy.squeue``
+* ``scrapy.stats``
+* ``scrapy.statscol``
+* ``scrapy.utils.decorator``
+
+See :ref:`module-relocations` for more information, or use suggestions
+from Scrapy 1.5.x deprecation warnings to update your code.
+
+Other deprecation removals:
+
+* Deprecated scrapy.interfaces.ISpiderManager is removed; please use
+ scrapy.interfaces.ISpiderLoader.
+* Deprecated ``CrawlerSettings`` class is removed (:gh:`3327`).
+* Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes
+ are removed (:gh:`3327`, :gh:`3359`).
+
+Other improvements, cleanups
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* All Scrapy tests now pass on Windows; Scrapy testing suite is executed
+ in a Windows environment on CI (:gh:`3315`).
+* Python 3.7 support (:gh:`3326`, :gh:`3150`, :gh:`3547`).
+* Testing and CI fixes (:gh:`3526`, :gh:`3538`, :gh:`3308`,
+ :gh:`3311`, :gh:`3309`, :gh:`3305`, :gh:`3210`, :gh:`3299`)
+* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name"
+ optional arguments (:gh:`3231`).
+* additional files are included to sdist (:gh:`3495`);
+* code style fixes (:gh:`3405`, :gh:`3304`);
+* unneeded .strip() call is removed (:gh:`3519`);
+* collections.deque is used to store MiddlewareManager methods instead
+ of a list (:gh:`3476`)
+
+.. _release-1.5.2:
+
+Scrapy 1.5.2 (2019-01-22)
+-------------------------
+
+* *Security bugfix*: Telnet console extension can be easily exploited by rogue
+ websites POSTing content to http://localhost:6023, we haven't found a way to
+ exploit it from Scrapy, but it is very easy to trick a browser to do so and
+ elevates the risk for local development environment.
+
+ *The fix is backward incompatible*, it enables telnet user-password
+ authentication by default with a random generated password. If you can't
+ upgrade right away, please consider setting :setting:`TELNETCONSOLE_PORT`
+ out of its default value.
+
+ See :ref:`telnet console <topics-telnetconsole>` documentation for more info
+
+* Backport CI build failure under GCE environment due to boto import error.
+
+.. _release-1.5.1:
+
+Scrapy 1.5.1 (2018-07-12)
+-------------------------
+
+This is a maintenance release with important bug fixes, but no new features:
+
+* ``O(N^2)`` gzip decompression issue which affected Python 3 and PyPy
+ is fixed (:gh:`3281`);
+* skipping of TLS validation errors is improved (:gh:`3166`);
+* Ctrl-C handling is fixed in Python 3.5+ (:gh:`3096`);
+* testing fixes (:gh:`3092`, :gh:`3263`);
+* documentation improvements (:gh:`3058`, :gh:`3059`, :gh:`3089`,
+ :gh:`3123`, :gh:`3127`, :gh:`3189`, :gh:`3224`, :gh:`3280`,
+ :gh:`3279`, :gh:`3201`, :gh:`3260`, :gh:`3284`, :gh:`3298`,
+ :gh:`3294`).
+
+
+.. _release-1.5.0:
+
+Scrapy 1.5.0 (2017-12-29)
+-------------------------
+
+This release brings small new features and improvements across the codebase.
+Some highlights:
+
+* Google Cloud Storage is supported in FilesPipeline and ImagesPipeline.
+* Crawling with proxy servers becomes more efficient, as connections
+ to proxies can be reused now.
+* Warnings, exception and logging messages are improved to make debugging
+ easier.
+* ``scrapy parse`` command now allows to set custom request meta via
+ ``--meta`` argument.
+* Compatibility with Python 3.6, PyPy and PyPy3 is improved;
+ PyPy and PyPy3 are now supported officially, by running tests on CI.
+* Better default handling of HTTP 308, 522 and 524 status codes.
+* Documentation is improved, as usual.
+
+Backward Incompatible Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Scrapy 1.5 drops support for Python 3.3.
+* Default Scrapy User-Agent now uses https link to scrapy.org (:gh:`2983`).
+ **This is technically backward-incompatible**; override
+ :setting:`USER_AGENT` if you relied on old value.
+* Logging of settings overridden by ``custom_settings`` is fixed;
+ **this is technically backward-incompatible** because the logger
+ changes from ``[scrapy.utils.log]`` to ``[scrapy.crawler]``. If you're
+ parsing Scrapy logs, please update your log parsers (:gh:`1343`).
+* LinkExtractor now ignores ``m4v`` extension by default, this is change
+ in behavior.
+* 522 and 524 status codes are added to ``RETRY_HTTP_CODES`` (:gh:`2851`)
+
+New features
+~~~~~~~~~~~~
+
+- Support ``<link>`` tags in ``Response.follow`` (:gh:`2785`)
+- Support for ``ptpython`` REPL (:gh:`2654`)
+- Google Cloud Storage support for FilesPipeline and ImagesPipeline
+ (:gh:`2923`).
+- New ``--meta`` option of the "scrapy parse" command allows to pass additional
+ request.meta (:gh:`2883`)
+- Populate spider variable when using ``shell.inspect_response`` (:gh:`2812`)
+- Handle HTTP 308 Permanent Redirect (:gh:`2844`)
+- Add 522 and 524 to ``RETRY_HTTP_CODES`` (:gh:`2851`)
+- Log versions information at startup (:gh:`2857`)
+- ``scrapy.mail.MailSender`` now works in Python 3 (it requires Twisted 17.9.0)
+- Connections to proxy servers are reused (:gh:`2743`)
+- Add template for a downloader middleware (:gh:`2755`)
+- Explicit message for NotImplementedError when parse callback not defined
+ (:gh:`2831`)
+- CrawlerProcess got an option to disable installation of root log handler
+ (:gh:`2921`)
+- LinkExtractor now ignores ``m4v`` extension by default
+- Better log messages for responses over :setting:`DOWNLOAD_WARNSIZE` and
+ :setting:`DOWNLOAD_MAXSIZE` limits (:gh:`2927`)
+- Show warning when a URL is put to ``Spider.allowed_domains`` instead of
+ a domain (:gh:`2250`).
+
+Bug fixes
+~~~~~~~~~
+
+- Fix logging of settings overridden by ``custom_settings``;
+ **this is technically backward-incompatible** because the logger
+ changes from ``[scrapy.utils.log]`` to ``[scrapy.crawler]``, so please
+ update your log parsers if needed (:gh:`1343`)
+- Default Scrapy User-Agent now uses https link to scrapy.org (:gh:`2983`).
+ **This is technically backward-incompatible**; override
+ :setting:`USER_AGENT` if you relied on old value.
+- Fix PyPy and PyPy3 test failures, support them officially
+ (:gh:`2793`, :gh:`2935`, :gh:`2990`, :gh:`3050`, :gh:`2213`,
+ :gh:`3048`)
+- Fix DNS resolver when ``DNSCACHE_ENABLED=False`` (:gh:`2811`)
+- Add ``cryptography`` for Debian Jessie tox test env (:gh:`2848`)
+- Add verification to check if Request callback is callable (:gh:`2766`)
+- Port ``extras/qpsclient.py`` to Python 3 (:gh:`2849`)
+- Use getfullargspec under the scenes for Python 3 to stop DeprecationWarning
+ (:gh:`2862`)
+- Update deprecated test aliases (:gh:`2876`)
+- Fix ``SitemapSpider`` support for alternate links (:gh:`2853`)
+
+Docs
+~~~~
+
+- Added missing bullet point for the ``AUTOTHROTTLE_TARGET_CONCURRENCY``
+ setting. (:gh:`2756`)
+- Update Contributing docs, document new support channels
+ (:gh:`2762`, :gh:`3038`)
+- Include references to Scrapy subreddit in the docs
+- Fix broken links; use ``https://`` for external links
+ (:gh:`2978`, :gh:`2982`, :gh:`2958`)
+- Document CloseSpider extension better (:gh:`2759`)
+- Use ``pymongo.collection.Collection.insert_one()`` in MongoDB example
+ (:gh:`2781`)
+- Spelling mistake and typos
+ (:gh:`2828`, :gh:`2837`, :gh:`2884`, :gh:`2924`)
+- Clarify ``CSVFeedSpider.headers`` documentation (:gh:`2826`)
+- Document ``DontCloseSpider`` exception and clarify ``spider_idle``
+ (:gh:`2791`)
+- Update "Releases" section in README (:gh:`2764`)
+- Fix rst syntax in ``DOWNLOAD_FAIL_ON_DATALOSS`` docs (:gh:`2763`)
+- Small fix in description of startproject arguments (:gh:`2866`)
+- Clarify data types in Response.body docs (:gh:`2922`)
+- Add a note about ``request.meta['depth']`` to DepthMiddleware docs (:gh:`2374`)
+- Add a note about ``request.meta['dont_merge_cookies']`` to CookiesMiddleware
+ docs (:gh:`2999`)
+- Up-to-date example of project structure (:gh:`2964`, :gh:`2976`)
+- A better example of ItemExporters usage (:gh:`2989`)
+- Document ``from_crawler`` methods for spider and downloader middlewares
+ (:gh:`3019`)
+
+.. _release-1.4.0:
+
+Scrapy 1.4.0 (2017-05-18)
+-------------------------
+
+Scrapy 1.4 does not bring that many breathtaking new features
+but quite a few handy improvements nonetheless.
+
+Scrapy now supports anonymous FTP sessions with customizable user and
+password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings.
+And if you're using Twisted version 17.1.0 or above, FTP is now available
+with Python 3.
+
+There's a new :meth:`response.follow <scrapy.http.TextResponse.follow>` method
+for creating requests; **it is now a recommended way to create Requests
+in Scrapy spiders**. This method makes it easier to write correct
+spiders; ``response.follow`` has several advantages over creating
+``scrapy.Request`` objects directly:
+
+* it handles relative URLs;
+* it works properly with non-ascii URLs on non-UTF8 pages;
+* in addition to absolute and relative URLs it supports Selectors;
+ for ``<a>`` elements it can also extract their href values.
+
+For example, instead of this::
+
+ for href in response.css('li.page a::attr(href)').extract():
+ url = response.urljoin(href)
+ yield scrapy.Request(url, self.parse, encoding=response.encoding)
+
+One can now write this::
+
+ for a in response.css('li.page a'):
+ yield response.follow(a, self.parse)
+
+Link extractors are also improved. They work similarly to what a regular
+modern browser would do: leading and trailing whitespace are removed
+from attributes (think ``href=" http://example.com"``) when building
+``Link`` objects. This whitespace-stripping also happens for ``action``
+attributes with ``FormRequest``.
+
+**Please also note that link extractors do not canonicalize URLs by default
+anymore.** This was puzzling users every now and then, and it's not what
+browsers do in fact, so we removed that extra transformation on extracted
+links.
+
+For those of you wanting more control on the ``Referer:`` header that Scrapy
+sends when following links, you can set your own ``Referrer Policy``.
+Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and
+blindly set it to the URL of the response that generated the HTTP request
+(which could leak information on your URL seeds).
+By default, Scrapy now behaves much like your regular browser does.
+And this policy is fully customizable with W3C standard values
+(or with something really custom of your own if you wish).
+See :setting:`REFERRER_POLICY` for details.
+
+To make Scrapy spiders easier to debug, Scrapy logs more stats by default
+in 1.4: memory usage stats, detailed retry stats, detailed HTTP error code
+stats. A similar change is that HTTP cache path is also visible in logs now.
+
+Last but not least, Scrapy now has the option to make JSON and XML items
+more human-readable, with newlines between items and even custom indenting
+offset, using the new :setting:`FEED_EXPORT_INDENT` setting.
+
+Enjoy! (Or read on for the rest of changes in this release.)
+
+Deprecations and Backward Incompatible Changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Default to ``canonicalize=False`` in
+ :class:`scrapy.linkextractors.LinkExtractor
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ (:gh:`2537`, fixes :gh:`1941` and :gh:`1982`):
+ **warning, this is technically backward-incompatible**
+- Enable memusage extension by default (:gh:`2539`, fixes :gh:`2187`);
+ **this is technically backward-incompatible** so please check if you have
+ any non-default ``MEMUSAGE_***`` options set.
+- ``EDITOR`` environment variable now takes precedence over ``EDITOR``
+ option defined in settings.py (:gh:`1829`); Scrapy default settings
+ no longer depend on environment variables. **This is technically a backward
+ incompatible change**.
+- ``Spider.make_requests_from_url`` is deprecated
+ (:gh:`1728`, fixes :gh:`1495`).
+
+New Features
+~~~~~~~~~~~~
+
+- Accept proxy credentials in :reqmeta:`proxy` request meta key (:gh:`2526`)
+- Support `brotli-compressed`_ content; requires optional `brotlipy`_
+ (:gh:`2535`)
+- New :ref:`response.follow <response-follow-example>` shortcut
+ for creating requests (:gh:`1940`)
+- Added ``flags`` argument and attribute to :class:`~scrapy.Request`
+ objects (:gh:`2047`)
+- Support Anonymous FTP (:gh:`2342`)
+- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/<reason>``
+ stats to :class:`RetryMiddleware <scrapy.downloadermiddlewares.retry.RetryMiddleware>`
+ (:gh:`2543`)
+- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/<status>``
+ stats to :class:`HttpErrorMiddleware <scrapy.spidermiddlewares.httperror.HttpErrorMiddleware>`
+ (:gh:`2566`)
+- Customizable :setting:`Referrer policy <REFERRER_POLICY>` in
+ :class:`RefererMiddleware <scrapy.spidermiddlewares.referer.RefererMiddleware>`
+ (:gh:`2306`)
+- New ``data:`` URI download handler (:gh:`2334`, fixes :gh:`2156`)
+- Log cache directory when HTTP Cache is used (:gh:`2611`, fixes :gh:`2604`)
+- Warn users when project contains duplicate spider names (fixes :gh:`2181`)
+- ``scrapy.utils.datatypes.CaselessDict`` now accepts ``Mapping`` instances and
+ not only dicts (:gh:`2646`)
+- :ref:`Media downloads <topics-media-pipeline>`, with
+ :class:`~scrapy.pipelines.files.FilesPipeline` or
+ :class:`~scrapy.pipelines.images.ImagesPipeline`, can now optionally handle
+ HTTP redirects using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting
+ (:gh:`2616`, fixes :gh:`2004`)
+- Accept non-complete responses from websites using a new
+ :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:gh:`2590`, fixes :gh:`2586`)
+- Optional pretty-printing of JSON and XML items via
+ :setting:`FEED_EXPORT_INDENT` setting (:gh:`2456`, fixes :gh:`1327`)
+- Allow dropping fields in ``FormRequest.from_response`` formdata when
+ ``None`` value is passed (:gh:`667`)
+- Per-request retry times with the new :reqmeta:`max_retry_times` meta key
+ (:gh:`2642`)
+- ``python -m scrapy`` as a more explicit alternative to ``scrapy`` command
+ (:gh:`2740`)
+
+.. _brotli-compressed: https://www.ietf.org/rfc/rfc7932.txt
+.. _brotlipy: https://github.com/python-hyper/brotlipy/
+
+Bug fixes
+~~~~~~~~~
+
+- LinkExtractor now strips leading and trailing whitespaces from attributes
+ (:gh:`2547`, fixes :gh:`1614`)
+- Properly handle whitespaces in action attribute in
+ :class:`~scrapy.FormRequest` (:gh:`2548`)
+- Buffer CONNECT response bytes from proxy until all HTTP headers are received
+ (:gh:`2495`, fixes :gh:`2491`)
+- FTP downloader now works on Python 3, provided you use Twisted>=17.1
+ (:gh:`2599`)
+- Use body to choose response type after decompressing content (:gh:`2393`,
+ fixes :gh:`2145`)
+- Always decompress ``Content-Encoding: gzip`` at :class:`HttpCompressionMiddleware
+ <scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware>` stage (:gh:`2391`)
+- Respect custom log level in ``Spider.custom_settings`` (:gh:`2581`,
+ fixes :gh:`1612`)
+- 'make htmlview' fix for macOS (:gh:`2661`)
+- Remove "commands" from the command list (:gh:`2695`)
+- Fix duplicate Content-Length header for POST requests with empty body (:gh:`2677`)
+- Properly cancel large downloads, i.e. above :setting:`DOWNLOAD_MAXSIZE` (:gh:`1616`)
+- ImagesPipeline: fixed processing of transparent PNG images with palette
+ (:gh:`2675`)
+
+Cleanups & Refactoring
+~~~~~~~~~~~~~~~~~~~~~~
+
+- Tests: remove temp files and folders (:gh:`2570`),
+ fixed ProjectUtilsTest on macOS (:gh:`2569`),
+ use portable pypy for Linux on Travis CI (:gh:`2710`)
+- Separate building request from ``_requests_to_follow`` in CrawlSpider (:gh:`2562`)
+- Remove “Python 3 progress” badge (:gh:`2567`)
+- Add a couple more lines to ``.gitignore`` (:gh:`2557`)
+- Remove bumpversion prerelease configuration (:gh:`2159`)
+- Add codecov.yml file (:gh:`2750`)
+- Set context factory implementation based on Twisted version (:gh:`2577`,
+ fixes :gh:`2560`)
+- Add omitted ``self`` arguments in default project middleware template (:gh:`2595`)
+- Remove redundant ``slot.add_request()`` call in ExecutionEngine (:gh:`2617`)
+- Catch more specific ``os.error`` exception in
+ ``scrapy.pipelines.files.FSFilesStore`` (:gh:`2644`)
+- Change "localhost" test server certificate (:gh:`2720`)
+- Remove unused ``MEMUSAGE_REPORT`` setting (:gh:`2576`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Binary mode is required for exporters (:gh:`2564`, fixes :gh:`2553`)
+- Mention issue with :meth:`.FormRequest.from_response` due to bug in lxml (:gh:`2572`)
+- Use single quotes uniformly in templates (:gh:`2596`)
+- Document :reqmeta:`ftp_user` and :reqmeta:`ftp_password` meta keys (:gh:`2587`)
+- Removed section on deprecated ``contrib/`` (:gh:`2636`)
+- Recommend Anaconda when installing Scrapy on Windows
+ (:gh:`2477`, fixes :gh:`2475`)
+- FAQ: rewrite note on Python 3 support on Windows (:gh:`2690`)
+- Rearrange selector sections (:gh:`2705`)
+- Remove ``__nonzero__`` from :class:`~scrapy.selector.SelectorList`
+ docs (:gh:`2683`)
+- Mention how to disable request filtering in documentation of
+ :setting:`DUPEFILTER_CLASS` setting (:gh:`2714`)
+- Add sphinx_rtd_theme to docs setup readme (:gh:`2668`)
+- Open file in text mode in JSON item writer example (:gh:`2729`)
+- Clarify ``allowed_domains`` example (:gh:`2670`)
+
+
+.. _release-1.3.3:
+
+Scrapy 1.3.3 (2017-03-10)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Make ``SpiderLoader`` raise ``ImportError`` again by default for missing
+ dependencies and wrong :setting:`SPIDER_MODULES`.
+ These exceptions were silenced as warnings since 1.3.0.
+ A new setting is introduced to toggle between warning or exception if needed ;
+ see :setting:`SPIDER_LOADER_WARN_ONLY` for details.
+
+.. _release-1.3.2:
+
+Scrapy 1.3.2 (2017-02-13)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Preserve request class when converting to/from dicts (utils.reqser) (:gh:`2510`).
+- Use consistent selectors for author field in tutorial (:gh:`2551`).
+- Fix TLS compatibility in Twisted 17+ (:gh:`2558`)
+
+.. _release-1.3.1:
+
+Scrapy 1.3.1 (2017-02-08)
+-------------------------
+
+New features
+~~~~~~~~~~~~
+
+- Support ``'True'`` and ``'False'`` string values for boolean settings (:gh:`2519`);
+ you can now do something like ``scrapy crawl myspider -s REDIRECT_ENABLED=False``.
+- Support kwargs with ``response.xpath()`` to use :ref:`XPath variables <topics-selectors-xpath-variables>`
+ and ad-hoc namespaces declarations ;
+ this requires at least Parsel v1.1 (:gh:`2457`).
+- Add support for Python 3.6 (:gh:`2485`).
+- Run tests on PyPy (warning: some tests still fail, so PyPy is not supported yet).
+
+Bug fixes
+~~~~~~~~~
+
+- Enforce ``DNS_TIMEOUT`` setting (:gh:`2496`).
+- Fix :command:`view` command ; it was a regression in v1.3.0 (:gh:`2503`).
+- Fix tests regarding ``*_EXPIRES settings`` with Files/Images pipelines (:gh:`2460`).
+- Fix name of generated pipeline class when using basic project template (:gh:`2466`).
+- Fix compatibility with Twisted 17+ (:gh:`2496`, :gh:`2528`).
+- Fix ``scrapy.Item`` inheritance on Python 3.6 (:gh:`2511`).
+- Enforce numeric values for components order in ``SPIDER_MIDDLEWARES``,
+ ``DOWNLOADER_MIDDLEWARES``, ``EXTENSIONS`` and ``SPIDER_CONTRACTS`` (:gh:`2420`).
+
+Documentation
+~~~~~~~~~~~~~
+
+- Reword Code of Conduct section and upgrade to Contributor Covenant v1.4
+ (:gh:`2469`).
+- Clarify that passing spider arguments converts them to spider attributes
+ (:gh:`2483`).
+- Document ``formid`` argument on ``FormRequest.from_response()`` (:gh:`2497`).
+- Add .rst extension to README files (:gh:`2507`).
+- Mention LevelDB cache storage backend (:gh:`2525`).
+- Use ``yield`` in sample callback code (:gh:`2533`).
+- Add note about HTML entities decoding with ``.re()/.re_first()`` (:gh:`1704`).
+- Typos (:gh:`2512`, :gh:`2534`, :gh:`2531`).
+
+Cleanups
+~~~~~~~~
+
+- Remove redundant check in ``MetaRefreshMiddleware`` (:gh:`2542`).
+- Faster checks in ``LinkExtractor`` for allow/deny patterns (:gh:`2538`).
+- Remove dead code supporting old Twisted versions (:gh:`2544`).
+
+
+.. _release-1.3.0:
+
+Scrapy 1.3.0 (2016-12-21)
+-------------------------
+
+This release comes rather soon after 1.2.2 for one main reason:
+it was found out that releases since 0.18 up to 1.2.2 (included) use
+some backported code from Twisted (``scrapy.xlib.tx.*``),
+even if newer Twisted modules are available.
+Scrapy now uses ``twisted.web.client`` and ``twisted.internet.endpoints`` directly.
+(See also cleanups below.)
+
+As it is a major change, we wanted to get the bug fix out quickly
+while not breaking any projects using the 1.2 series.
+
+New Features
+~~~~~~~~~~~~
+
+- ``MailSender`` now accepts single strings as values for ``to`` and ``cc``
+ arguments (:gh:`2272`)
+- ``scrapy fetch url``, ``scrapy shell url`` and ``fetch(url)`` inside
+ Scrapy shell now follow HTTP redirections by default (:gh:`2290`);
+ See :command:`fetch` and :command:`shell` for details.
+- ``HttpErrorMiddleware`` now logs errors with ``INFO`` level instead of ``DEBUG``;
+ this is technically **backward incompatible** so please check your log parsers.
+- By default, logger names now use a long-form path, e.g. ``[scrapy.extensions.logstats]``,
+ instead of the shorter "top-level" variant of prior releases (e.g. ``[scrapy]``);
+ this is **backward incompatible** if you have log parsers expecting the short
+ logger name part. You can switch back to short logger names using :setting:`LOG_SHORT_NAMES`
+ set to ``True``.
+
+Dependencies & Cleanups
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- Scrapy now requires Twisted >= 13.1 which is the case for many Linux
+ distributions already.
+- As a consequence, we got rid of ``scrapy.xlib.tx.*`` modules, which
+ copied some of Twisted code for users stuck with an "old" Twisted version
+- ``ChunkedTransferMiddleware`` is deprecated and removed from the default
+ downloader middlewares.
+
+.. _release-1.2.3:
+
+Scrapy 1.2.3 (2017-03-03)
+-------------------------
+
+- Packaging fix: disallow unsupported Twisted versions in setup.py
+
+
+.. _release-1.2.2:
+
+Scrapy 1.2.2 (2016-12-06)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Fix a cryptic traceback when a pipeline fails on ``open_spider()`` (:gh:`2011`)
+- Fix embedded IPython shell variables (fixing :gh:`396` that re-appeared
+ in 1.2.0, fixed in :gh:`2418`)
+- A couple of patches when dealing with robots.txt:
+
+ - handle (non-standard) relative sitemap URLs (:gh:`2390`)
+ - handle non-ASCII URLs and User-Agents in Python 2 (:gh:`2373`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Document ``"download_latency"`` key in ``Request``'s ``meta`` dict (:gh:`2033`)
+- Remove page on (deprecated & unsupported) Ubuntu packages from ToC (:gh:`2335`)
+- A few fixed typos (:gh:`2346`, :gh:`2369`, :gh:`2369`, :gh:`2380`)
+ and clarifications (:gh:`2354`, :gh:`2325`, :gh:`2414`)
+
+Other changes
+~~~~~~~~~~~~~
+
+- Advertize `conda-forge`_ as Scrapy's official conda channel (:gh:`2387`)
+- More helpful error messages when trying to use ``.css()`` or ``.xpath()``
+ on non-Text Responses (:gh:`2264`)
+- ``startproject`` command now generates a sample ``middlewares.py`` file (:gh:`2335`)
+- Add more dependencies' version info in ``scrapy version`` verbose output (:gh:`2404`)
+- Remove all ``*.pyc`` files from source distribution (:gh:`2386`)
+
+.. _conda-forge: https://anaconda.org/conda-forge/scrapy
+
+
+.. _release-1.2.1:
+
+Scrapy 1.2.1 (2016-10-21)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Include OpenSSL's more permissive default ciphers when establishing
+ TLS/SSL connections (:gh:`2314`).
+- Fix "Location" HTTP header decoding on non-ASCII URL redirects (:gh:`2321`).
+
+Documentation
+~~~~~~~~~~~~~
+
+- Fix JsonWriterPipeline example (:gh:`2302`).
+- Various notes: :gh:`2330` on spider names,
+ :gh:`2329` on middleware methods processing order,
+ :gh:`2327` on getting multi-valued HTTP headers as lists.
+
+Other changes
+~~~~~~~~~~~~~
+
+- Removed ``www.`` from ``start_urls`` in built-in spider templates (:gh:`2299`).
+
+
+.. _release-1.2.0:
+
+Scrapy 1.2.0 (2016-10-03)
+-------------------------
+
+New Features
+~~~~~~~~~~~~
+
+- New :setting:`FEED_EXPORT_ENCODING` setting to customize the encoding
+ used when writing items to a file.
+ This can be used to turn off ``\uXXXX`` escapes in JSON output.
+ This is also useful for those wanting something else than UTF-8
+ for XML or CSV output (:gh:`2034`).
+- ``startproject`` command now supports an optional destination directory
+ to override the default one based on the project name (:gh:`2005`).
+- New :setting:`SCHEDULER_DEBUG` setting to log requests serialization
+ failures (:gh:`1610`).
+- JSON encoder now supports serialization of ``set`` instances (:gh:`2058`).
+- Interpret ``application/json-amazonui-streaming`` as ``TextResponse`` (:gh:`1503`).
+- ``scrapy`` is imported by default when using shell tools (:command:`shell`,
+ :ref:`inspect_response <topics-shell-inspect-response>`) (:gh:`2248`).
+
+Bug fixes
+~~~~~~~~~
+
+- DefaultRequestHeaders middleware now runs before UserAgent middleware
+ (:gh:`2088`). **Warning: this is technically backward incompatible**,
+ though we consider this a bug fix.
+- HTTP cache extension and plugins that use the ``.scrapy`` data directory now
+ work outside projects (:gh:`1581`). **Warning: this is technically
+ backward incompatible**, though we consider this a bug fix.
+- ``Selector`` does not allow passing both ``response`` and ``text`` anymore
+ (:gh:`2153`).
+- Fixed logging of wrong callback name with ``scrapy parse`` (:gh:`2169`).
+- Fix for an odd gzip decompression bug (:gh:`1606`).
+- Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse <parse>`
+ (:gh:`2225`).
+- Fix for invalid JSON and XML files when spider yields no items (:gh:`872`).
+- Implement ``flush()`` for ``StreamLogger`` avoiding a warning in logs (:gh:`2125`).
+
+Refactoring
+~~~~~~~~~~~
+
+- ``canonicalize_url`` has been moved to `w3lib.url`_ (:gh:`2168`).
+
+.. _w3lib.url: https://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.url.canonicalize_url
+
+Tests & Requirements
+~~~~~~~~~~~~~~~~~~~~
+
+Scrapy's new requirements baseline is Debian 8 "Jessie". It was previously
+Ubuntu 12.04 Precise.
+What this means in practice is that we run continuous integration tests
+with these (main) packages versions at a minimum:
+Twisted 14.0, pyOpenSSL 0.14, lxml 3.4.
+
+Scrapy may very well work with older versions of these packages
+(the code base still has switches for older Twisted versions for example)
+but it is not guaranteed (because it's not tested anymore).
+
+Documentation
+~~~~~~~~~~~~~
+
+- Grammar fixes: :gh:`2128`, :gh:`1566`.
+- Download stats badge removed from README (:gh:`2160`).
+- New Scrapy :ref:`architecture diagram <topics-architecture>` (:gh:`2165`).
+- Updated ``Response`` parameters documentation (:gh:`2197`).
+- Reworded misleading ``RANDOMIZE_DOWNLOAD_DELAY`` description (:gh:`2190`).
+- Add StackOverflow as a support channel (:gh:`2257`).
+
+.. _release-1.1.4:
+
+Scrapy 1.1.4 (2017-03-03)
+-------------------------
+
+- Packaging fix: disallow unsupported Twisted versions in setup.py
+
+.. _release-1.1.3:
+
+Scrapy 1.1.3 (2016-09-22)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Class attributes for subclasses of ``ImagesPipeline`` and ``FilesPipeline``
+ work as they did before 1.1.1 (:gh:`2243`, fixes :gh:`2198`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- :ref:`Overview <intro-overview>` and :ref:`tutorial <intro-tutorial>`
+ rewritten to use http://toscrape.com websites
+ (:gh:`2236`, :gh:`2249`, :gh:`2252`).
+
+.. _release-1.1.2:
+
+Scrapy 1.1.2 (2016-08-18)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Introduce a missing :setting:`IMAGES_STORE_S3_ACL` setting to override
+ the default ACL policy in ``ImagesPipeline`` when uploading images to S3
+ (note that default ACL policy is "private" -- instead of "public-read" --
+ since Scrapy 1.1.0)
+- :setting:`IMAGES_EXPIRES` default value set back to 90
+ (the regression was introduced in 1.1.1)
+
+.. _release-1.1.1:
+
+Scrapy 1.1.1 (2016-07-13)
+-------------------------
+
+Bug fixes
+~~~~~~~~~
+
+- Add "Host" header in CONNECT requests to HTTPS proxies (:gh:`2069`)
+- Use response ``body`` when choosing response class
+ (:gh:`2001`, fixes :gh:`2000`)
+- Do not fail on canonicalizing URLs with wrong netlocs
+ (:gh:`2038`, fixes :gh:`2010`)
+- a few fixes for ``HttpCompressionMiddleware`` (and ``SitemapSpider``):
+
+ - Do not decode HEAD responses (:gh:`2008`, fixes :gh:`1899`)
+ - Handle charset parameter in gzip Content-Type header
+ (:gh:`2050`, fixes :gh:`2049`)
+ - Do not decompress gzip octet-stream responses
+ (:gh:`2065`, fixes :gh:`2063`)
+
+- Catch (and ignore with a warning) exception when verifying certificate
+ against IP-address hosts (:gh:`2094`, fixes :gh:`2092`)
+- Make ``FilesPipeline`` and ``ImagesPipeline`` backward compatible again
+ regarding the use of legacy class attributes for customization
+ (:gh:`1989`, fixes :gh:`1985`)
+
+
+New features
+~~~~~~~~~~~~
+
+- Enable genspider command outside project folder (:gh:`2052`)
+- Retry HTTPS CONNECT ``TunnelError`` by default (:gh:`1974`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+- ``FEED_TEMPDIR`` setting at lexicographical position (:commit:`9b3c72c`)
+- Use idiomatic ``.extract_first()`` in overview (:gh:`1994`)
+- Update years in copyright notice (:commit:`c2c8036`)
+- Add information and example on errbacks (:gh:`1995`)
+- Use "url" variable in downloader middleware example (:gh:`2015`)
+- Grammar fixes (:gh:`2054`, :gh:`2120`)
+- New FAQ entry on using BeautifulSoup in spider callbacks (:gh:`2048`)
+- Add notes about Scrapy not working on Windows with Python 3 (:gh:`2060`)
+- Encourage complete titles in pull requests (:gh:`2026`)
+
+Tests
+~~~~~
+
+- Upgrade py.test requirement on Travis CI and Pin pytest-cov to 2.2.1 (:gh:`2095`)
+
+.. _release-1.1.0:
+
+Scrapy 1.1.0 (2016-05-11)
+-------------------------
+
+This 1.1 release brings a lot of interesting features and bug fixes:
+
+- Scrapy 1.1 has beta Python 3 support (requires Twisted >= 15.5). See
+ :ref:`news_betapy3` for more details and some limitations.
+- Hot new features:
+
+ - Item loaders now support nested loaders (:gh:`1467`).
+ - ``FormRequest.from_response`` improvements (:gh:`1382`, :gh:`1137`).
+ - Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improved
+ AutoThrottle docs (:gh:`1324`).
+ - Added ``response.text`` to get body as unicode (:gh:`1730`).
+ - Anonymous S3 connections (:gh:`1358`).
+ - Deferreds in downloader middlewares (:gh:`1473`). This enables better
+ robots.txt handling (:gh:`1471`).
+ - HTTP caching now follows RFC2616 more closely, added settings
+ :setting:`HTTPCACHE_ALWAYS_STORE` and
+ :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:gh:`1151`).
+ - Selectors were extracted to the parsel_ library (:gh:`1409`). This means
+ you can use Scrapy Selectors without Scrapy and also upgrade the
+ selectors engine without needing to upgrade Scrapy.
+ - HTTPS downloader now does TLS protocol negotiation by default,
+ instead of forcing TLS 1.0. You can also set the SSL/TLS method
+ using the new ``DOWNLOADER_CLIENT_TLS_METHOD`` setting.
+
+- These bug fixes may require your attention:
+
+ - Don't retry bad requests (HTTP 400) by default (:gh:`1289`).
+ If you need the old behavior, add ``400`` to :setting:`RETRY_HTTP_CODES`.
+ - Fix shell files argument handling (:gh:`1710`, :gh:`1550`).
+ If you try ``scrapy shell index.html`` it will try to load the URL ``http://index.html``,
+ use ``scrapy shell ./index.html`` to load a local file.
+ - Robots.txt compliance is now enabled by default for newly-created projects
+ (:gh:`1724`). Scrapy will also wait for robots.txt to be downloaded
+ before proceeding with the crawl (:gh:`1735`). If you want to disable
+ this behavior, update :setting:`ROBOTSTXT_OBEY` in ``settings.py`` file
+ after creating a new project.
+ - Exporters now work on unicode, instead of bytes by default (:gh:`1080`).
+ If you use :class:`~scrapy.exporters.PythonItemExporter`, you may want to
+ update your code to disable binary mode which is now deprecated.
+ - Accept XML node names containing dots as valid (:gh:`1533`).
+ - When uploading files or images to S3 (with ``FilesPipeline`` or
+ ``ImagesPipeline``), the default ACL policy is now "private" instead
+ of "public" **Warning: backward incompatible!**.
+ You can use :setting:`FILES_STORE_S3_ACL` to change it.
+ - We've reimplemented ``canonicalize_url()`` for more correct output,
+ especially for URLs with non-ASCII characters (:gh:`1947`).
+ This could change link extractors output compared to previous Scrapy versions.
+ This may also invalidate some cache entries you could still have from pre-1.1 runs.
+ **Warning: backward incompatible!**.
+
+Keep reading for more details on other improvements and bug fixes.
+
+.. _news_betapy3:
+
+Beta Python 3 Support
+~~~~~~~~~~~~~~~~~~~~~
+
+We have been hard at work to make Scrapy run on Python 3. As a result, now
+you can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some
+features are still missing (and some may never be ported).
+
+
+Almost all builtin extensions/middlewares are expected to work.
+However, we are aware of some limitations in Python 3:
+
+- Scrapy does not work on Windows with Python 3
+- Sending emails is not supported
+- FTP download handler is not supported
+- Telnet console is not supported
+
+Additional New Features and Enhancements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Scrapy now has a `Code of Conduct`_ (:gh:`1681`).
+- Command line tool now has completion for zsh (:gh:`934`).
+- Improvements to ``scrapy shell``:
+
+ - Support for bpython and configure preferred Python shell via
+ ``SCRAPY_PYTHON_SHELL`` (:gh:`1100`, :gh:`1444`).
+ - Support URLs without scheme (:gh:`1498`)
+ **Warning: backward incompatible!**
+ - Bring back support for relative file path (:gh:`1710`, :gh:`1550`).
+
+- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check
+ interval (:gh:`1282`).
+- Download handlers are now lazy-loaded on first request using their
+ scheme (:gh:`1390`, :gh:`1421`).
+- HTTPS download handlers do not force TLS 1.0 anymore; instead,
+ OpenSSL's ``SSLv23_method()/TLS_method()`` is used allowing to try
+ negotiating with the remote hosts the highest TLS protocol version
+ it can (:gh:`1794`, :gh:`1629`).
+- ``RedirectMiddleware`` now skips the status codes from
+ ``handle_httpstatus_list`` on spider attribute
+ or in ``Request``'s ``meta`` key (:gh:`1334`, :gh:`1364`,
+ :gh:`1447`).
+- Form submission:
+
+ - now works with ``<button>`` elements too (:gh:`1469`).
+ - an empty string is now used for submit buttons without a value
+ (:gh:`1472`)
+
+- Dict-like settings now have per-key priorities
+ (:gh:`1135`, :gh:`1149` and :gh:`1586`).
+- Sending non-ASCII emails (:gh:`1662`)
+- ``CloseSpider`` and ``SpiderState`` extensions now get disabled if no relevant
+ setting is set (:gh:`1723`, :gh:`1725`).
+- Added method ``ExecutionEngine.close`` (:gh:`1423`).
+- Added method ``CrawlerRunner.create_crawler`` (:gh:`1528`).
+- Scheduler priority queue can now be customized via
+ :setting:`SCHEDULER_PRIORITY_QUEUE` (:gh:`1822`).
+- ``.pps`` links are now ignored by default in link extractors (:gh:`1835`).
+- temporary data folder for FTP and S3 feed storages can be customized
+ using a new :setting:`FEED_TEMPDIR` setting (:gh:`1847`).
+- ``FilesPipeline`` and ``ImagesPipeline`` settings are now instance attributes
+ instead of class attributes, enabling spider-specific behaviors (:gh:`1891`).
+- ``JsonItemExporter`` now formats opening and closing square brackets
+ on their own line (first and last lines of output file) (:gh:`1950`).
+- If available, ``botocore`` is used for ``S3FeedStorage``, ``S3DownloadHandler``
+ and ``S3FilesStore`` (:gh:`1761`, :gh:`1883`).
+- Tons of documentation updates and related fixes (:gh:`1291`, :gh:`1302`,
+ :gh:`1335`, :gh:`1683`, :gh:`1660`, :gh:`1642`, :gh:`1721`,
+ :gh:`1727`, :gh:`1879`).
+- Other refactoring, optimizations and cleanup (:gh:`1476`, :gh:`1481`,
+ :gh:`1477`, :gh:`1315`, :gh:`1290`, :gh:`1750`, :gh:`1881`).
+
+.. _Code of Conduct: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
+
+
+Deprecations and Removals
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Added ``to_bytes`` and ``to_unicode``, deprecated ``str_to_unicode`` and
+ ``unicode_to_str`` functions (:gh:`778`).
+- ``binary_is_text`` is introduced, to replace use of ``isbinarytext``
+ (but with inverse return value) (:gh:`1851`)
+- The ``optional_features`` set has been removed (:gh:`1359`).
+- The ``--lsprof`` command line option has been removed (:gh:`1689`).
+ **Warning: backward incompatible**, but doesn't break user code.
+- The following datatypes were deprecated (:gh:`1720`):
+
+ + ``scrapy.utils.datatypes.MultiValueDictKeyError``
+ + ``scrapy.utils.datatypes.MultiValueDict``
+ + ``scrapy.utils.datatypes.SiteNode``
+
+- The previously bundled ``scrapy.xlib.pydispatch`` library was deprecated and
+ replaced by `pydispatcher <https://pypi.org/project/PyDispatcher/>`_.
+
+
+Relocations
+~~~~~~~~~~~
+
+- ``telnetconsole`` was relocated to ``extensions/`` (:gh:`1524`).
+
+ + Note: telnet is not enabled on Python 3
+ (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)
+
+
+Bugfixes
+~~~~~~~~
+
+- Scrapy does not retry requests that got a ``HTTP 400 Bad Request``
+ response anymore (:gh:`1289`). **Warning: backward incompatible!**
+- Support empty password for http_proxy config (:gh:`1274`).
+- Interpret ``application/x-json`` as ``TextResponse`` (:gh:`1333`).
+- Support link rel attribute with multiple values (:gh:`1201`).
+- Fixed ``scrapy.FormRequest.from_response`` when there is a ``<base>``
+ tag (:gh:`1564`).
+- Fixed :setting:`TEMPLATES_DIR` handling (:gh:`1575`).
+- Various ``FormRequest`` fixes (:gh:`1595`, :gh:`1596`, :gh:`1597`).
+- Makes ``_monkeypatches`` more robust (:gh:`1634`).
+- Fixed bug on ``XMLItemExporter`` with non-string fields in
+ items (:gh:`1738`).
+- Fixed startproject command in macOS (:gh:`1635`).
+- Fixed :class:`~scrapy.exporters.PythonItemExporter` and CSVExporter for
+ non-string item types (:gh:`1737`).
+- Various logging related fixes (:gh:`1294`, :gh:`1419`, :gh:`1263`,
+ :gh:`1624`, :gh:`1654`, :gh:`1722`, :gh:`1726` and :gh:`1303`).
+- Fixed bug in ``utils.template.render_templatefile()`` (:gh:`1212`).
+- sitemaps extraction from ``robots.txt`` is now case-insensitive (:gh:`1902`).
+- HTTPS+CONNECT tunnels could get mixed up when using multiple proxies
+ to same remote host (:gh:`1912`).
+
+.. _release-1.0.7:
+
+Scrapy 1.0.7 (2017-03-03)
+-------------------------
+
+- Packaging fix: disallow unsupported Twisted versions in setup.py
+
+.. _release-1.0.6:
+
+Scrapy 1.0.6 (2016-05-04)
+-------------------------
+
+- FIX: RetryMiddleware is now robust to non-standard HTTP status codes (:gh:`1857`)
+- FIX: Filestorage HTTP cache was checking wrong modified time (:gh:`1875`)
+- DOC: Support for Sphinx 1.4+ (:gh:`1893`)
+- DOC: Consistency in selectors examples (:gh:`1869`)
+
+.. _release-1.0.5:
+
+Scrapy 1.0.5 (2016-02-04)
+-------------------------
+
+- FIX: [Backport] Ignore bogus links in LinkExtractors (fixes :gh:`907`, :commit:`108195e`)
+- TST: Changed buildbot makefile to use 'pytest' (:commit:`1f3d90a`)
+- DOC: Fixed typos in tutorial and media-pipeline (:commit:`808a9ea` and :commit:`803bd87`)
+- DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`)
+
+.. _release-1.0.4:
+
+Scrapy 1.0.4 (2015-12-30)
+-------------------------
+
+- Ignoring xlib/tx folder, depending on Twisted version. (:commit:`7dfa979`)
+- Run on new travis-ci infra (:commit:`6e42f0b`)
+- Spelling fixes (:commit:`823a1cc`)
+- escape nodename in xmliter regex (:commit:`da3c155`)
+- test xml nodename with dots (:commit:`4418fc3`)
+- TST don't use broken Pillow version in tests (:commit:`a55078c`)
+- disable log on version command. closes #1426 (:commit:`86fc330`)
+- disable log on startproject command (:commit:`db4c9fe`)
+- Add PyPI download stats badge (:commit:`df2b944`)
+- don't run tests twice on Travis if a PR is made from a scrapy/scrapy branch (:commit:`a83ab41`)
+- Add Python 3 porting status badge to the README (:commit:`73ac80d`)
+- fixed RFPDupeFilter persistence (:commit:`97d080e`)
+- TST a test to show that dupefilter persistence is not working (:commit:`97f2fb3`)
+- explicit close file on file:// scheme handler (:commit:`d9b4850`)
+- Disable dupefilter in shell (:commit:`c0d0734`)
+- DOC: Add captions to toctrees which appear in sidebar (:commit:`aa239ad`)
+- DOC Removed pywin32 from install instructions as it's already declared as dependency. (:commit:`10eb400`)
+- Added installation notes about using Conda for Windows and other OSes. (:commit:`1c3600a`)
+- Fixed minor grammar issues. (:commit:`7f4ddd5`)
+- fixed a typo in the documentation. (:commit:`b71f677`)
+- Version 1 now exists (:commit:`5456c0e`)
+- fix another invalid xpath error (:commit:`0a1366e`)
+- fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst (:commit:`ca8d60f`)
+- Typos corrections (:commit:`7067117`)
+- fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware (:commit:`32f115c`)
+- Add note to Ubuntu install section about Debian compatibility (:commit:`23fda69`)
+- Replace alternative macOS install workaround with virtualenv (:commit:`98b63ee`)
+- Reference Homebrew's homepage for installation instructions (:commit:`1925db1`)
+- Add oldest supported tox version to contributing docs (:commit:`5d10d6d`)
+- Note in install docs about pip being already included in python>=2.7.9 (:commit:`85c980e`)
+- Add non-python dependencies to Ubuntu install section in the docs (:commit:`fbd010d`)
+- Add macOS installation section to docs (:commit:`d8f4cba`)
+- DOC(ENH): specify path to rtd theme explicitly (:commit:`de73b1a`)
+- minor: scrapy.Spider docs grammar (:commit:`1ddcc7b`)
+- Make common practices sample code match the comments (:commit:`1b85bcf`)
+- nextcall repetitive calls (heartbeats). (:commit:`55f7104`)
+- Backport fix compatibility with Twisted 15.4.0 (:commit:`b262411`)
+- pin pytest to 2.7.3 (:commit:`a6535c2`)
+- Merge pull request #1512 from mgedmin/patch-1 (:commit:`8876111`)
+- Merge pull request #1513 from mgedmin/patch-2 (:commit:`5d4daf8`)
+- Typo (:commit:`f8d0682`)
+- Fix list formatting (:commit:`5f83a93`)
+- fix Scrapy squeue tests after recent changes to queuelib (:commit:`3365c01`)
+- Merge pull request #1475 from rweindl/patch-1 (:commit:`2d688cd`)
+- Update tutorial.rst (:commit:`fbc1f25`)
+- Merge pull request #1449 from rhoekman/patch-1 (:commit:`7d6538c`)
+- Small grammatical change (:commit:`8752294`)
+- Add openssl version to version command (:commit:`13c45ac`)
+
+.. _release-1.0.3:
+
+Scrapy 1.0.3 (2015-08-11)
+-------------------------
+
+- add service_identity to Scrapy install_requires (:commit:`cbc2501`)
+- Workaround for travis#296 (:commit:`66af9cd`)
+
+.. _release-1.0.2:
+
+Scrapy 1.0.2 (2015-08-06)
+-------------------------
+
+- Twisted 15.3.0 does not raises PicklingError serializing lambda functions (:commit:`b04dd7d`)
+- Minor method name fix (:commit:`6f85c7f`)
+- minor: scrapy.Spider grammar and clarity (:commit:`9c9d2e0`)
+- Put a blurb about support channels in CONTRIBUTING (:commit:`c63882b`)
+- Fixed typos (:commit:`a9ae7b0`)
+- Fix doc reference. (:commit:`7c8a4fe`)
+
+.. _release-1.0.1:
+
+Scrapy 1.0.1 (2015-07-01)
+-------------------------
+
+- Unquote request path before passing to FTPClient, it already escape paths (:commit:`cc00ad2`)
+- include tests/ to source distribution in MANIFEST.in (:commit:`eca227e`)
+- DOC Fix SelectJmes documentation (:commit:`b8567bc`)
+- DOC Bring Ubuntu and Archlinux outside of Windows subsection (:commit:`392233f`)
+- DOC remove version suffix from Ubuntu package (:commit:`5303c66`)
+- DOC Update release date for 1.0 (:commit:`c89fa29`)
+
+.. _release-1.0.0:
+
+Scrapy 1.0.0 (2015-06-19)
+-------------------------
+
+You will find a lot of new features and bugfixes in this major release. Make
+sure to check our updated :ref:`overview <intro-overview>` to get a glance of
+some of the changes, along with our brushed :ref:`tutorial <intro-tutorial>`.
+
+Support for returning dictionaries in spiders
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Declaring and returning Scrapy Items is no longer necessary to collect the
+scraped data from your spider, you can now return explicit dictionaries
+instead.
+
+*Classic version*
+
+::
+
+ class MyItem(scrapy.Item):
+ url = scrapy.Field()
+
+ class MySpider(scrapy.Spider):
+ def parse(self, response):
+ return MyItem(url=response.url)
+
+*New version*
+
+::
+
+ class MySpider(scrapy.Spider):
+ def parse(self, response):
+ return {'url': response.url}
+
+Per-spider settings (GSoC 2014)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Last Google Summer of Code project accomplished an important redesign of the
+mechanism used for populating settings, introducing explicit priorities to
+override any given setting. As an extension of that goal, we included a new
+level of priority for settings that act exclusively for a single spider,
+allowing them to redefine project settings.
+
+Start using it by defining a :attr:`~scrapy.spiders.Spider.custom_settings`
+class variable in your spider::
+
+ class MySpider(scrapy.Spider):
+ custom_settings = {
+ "DOWNLOAD_DELAY": 5.0,
+ "RETRY_ENABLED": False,
+ }
+
+Read more about settings population: :ref:`topics-settings`
+
+Python Logging
+~~~~~~~~~~~~~~
+
+Scrapy 1.0 has moved away from Twisted logging to support Python built in’s
+as default logging system. We’re maintaining backward compatibility for most
+of the old custom interface to call logging functions, but you’ll get
+warnings to switch to the Python logging API entirely.
+
+*Old version*
+
+::
+
+ from scrapy import log
+ log.msg('MESSAGE', log.INFO)
+
+*New version*
+
+::
+
+ import logging
+ logging.info('MESSAGE')
+
+Logging with spiders remains the same, but on top of the
+:meth:`~scrapy.spiders.Spider.log` method you’ll have access to a custom
+:attr:`~scrapy.spiders.Spider.logger` created for the spider to issue log
+events:
+
+::
+
+ class MySpider(scrapy.Spider):
+ def parse(self, response):
+ self.logger.info('Response received')
+
+Read more in the logging documentation: :ref:`topics-logging`
+
+Crawler API refactoring (GSoC 2014)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Another milestone for last Google Summer of Code was a refactoring of the
+internal API, seeking a simpler and easier usage. Check new core interface
+in: :ref:`topics-api`
+
+A common situation where you will face these changes is while running Scrapy
+from scripts. Here’s a quick example of how to run a Spider manually with the
+new API:
+
+::
+
+ from scrapy.crawler import CrawlerProcess
+
+ process = CrawlerProcess({
+ 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
+ })
+ process.crawl(MySpider)
+ process.start()
+
+Bear in mind this feature is still under development and its API may change
+until it reaches a stable status.
+
+See more examples for scripts running Scrapy: :ref:`topics-practices`
+
+.. _module-relocations:
+
+Module Relocations
+~~~~~~~~~~~~~~~~~~
+
+There’s been a large rearrangement of modules trying to improve the general
+structure of Scrapy. Main changes were separating various subpackages into
+new projects and dissolving both ``scrapy.contrib`` and ``scrapy.contrib_exp``
+into top level packages. Backward compatibility was kept among internal
+relocations, while importing deprecated modules expect warnings indicating
+their new place.
+
+Full list of relocations
+************************
+
+Outsourced packages
+
+.. note::
+ These extensions went through some minor changes, e.g. some setting names
+ were changed. Please check the documentation in each new repository to
+ get familiar with the new usage.
+
++-------------------------------------+-------------------------------------+
+| Old location | New location |
++=====================================+=====================================+
+| scrapy.commands.deploy | `scrapyd-client <https://github.com |
+| | /scrapy/scrapyd-client>`_ |
+| | (See other alternatives here: |
+| | :ref:`topics-deploy`) |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.djangoitem | `scrapy-djangoitem <https://github. |
+| | com/scrapy-plugins/scrapy-djangoite |
+| | m>`_ |
++-------------------------------------+-------------------------------------+
+| scrapy.webservice | `scrapy-jsonrpc <https://github.com |
+| | /scrapy-plugins/scrapy-jsonrpc>`_ |
++-------------------------------------+-------------------------------------+
+
+``scrapy.contrib_exp`` and ``scrapy.contrib`` dissolutions
+
++-------------------------------------+-------------------------------------+
+| Old location | New location |
++=====================================+=====================================+
+| scrapy.contrib\_exp.downloadermidd\ | scrapy.downloadermiddlewares.decom\ |
+| leware.decompression | pression |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib\_exp.iterators | scrapy.utils.iterators |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.downloadermiddleware | scrapy.downloadermiddlewares |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.exporter | scrapy.exporters |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.linkextractors | scrapy.linkextractors |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.loader | scrapy.loader |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.loader.processor | scrapy.loader.processors |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.pipeline | scrapy.pipelines |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.spidermiddleware | scrapy.spidermiddlewares |
++-------------------------------------+-------------------------------------+
+| scrapy.contrib.spiders | scrapy.spiders |
++-------------------------------------+-------------------------------------+
+| * scrapy.contrib.closespider | scrapy.extensions.\* |
+| * scrapy.contrib.corestats | |
+| * scrapy.contrib.debug | |
+| * scrapy.contrib.feedexport | |
+| * scrapy.contrib.httpcache | |
+| * scrapy.contrib.logstats | |
+| * scrapy.contrib.memdebug | |
+| * scrapy.contrib.memusage | |
+| * scrapy.contrib.spiderstate | |
+| * scrapy.contrib.statsmailer | |
+| * scrapy.contrib.throttle | |
++-------------------------------------+-------------------------------------+
+
+Plural renames and Modules unification
+
++-------------------------------------+-------------------------------------+
+| Old location | New location |
++=====================================+=====================================+
+| scrapy.command | scrapy.commands |
++-------------------------------------+-------------------------------------+
+| scrapy.dupefilter | scrapy.dupefilters |
++-------------------------------------+-------------------------------------+
+| scrapy.linkextractor | scrapy.linkextractors |
++-------------------------------------+-------------------------------------+
+| scrapy.spider | scrapy.spiders |
++-------------------------------------+-------------------------------------+
+| scrapy.squeue | scrapy.squeues |
++-------------------------------------+-------------------------------------+
+| scrapy.statscol | scrapy.statscollectors |
++-------------------------------------+-------------------------------------+
+| scrapy.utils.decorator | scrapy.utils.decorators |
++-------------------------------------+-------------------------------------+
+
+Class renames
+
++-------------------------------------+-------------------------------------+
+| Old location | New location |
++=====================================+=====================================+
+| scrapy.spidermanager.SpiderManager | scrapy.spiderloader.SpiderLoader |
++-------------------------------------+-------------------------------------+
+
+Settings renames
+
++-------------------------------------+-------------------------------------+
+| Old location | New location |
++=====================================+=====================================+
+| SPIDER\_MANAGER\_CLASS | SPIDER\_LOADER\_CLASS |
++-------------------------------------+-------------------------------------+
+
+Changelog
+~~~~~~~~~
+
+New Features and Enhancements
+
+- Python logging (:gh:`1060`, :gh:`1235`, :gh:`1236`, :gh:`1240`,
+ :gh:`1259`, :gh:`1278`, :gh:`1286`)
+- FEED_EXPORT_FIELDS option (:gh:`1159`, :gh:`1224`)
+- Dns cache size and timeout options (:gh:`1132`)
+- support namespace prefix in xmliter_lxml (:gh:`963`)
+- Reactor threadpool max size setting (:gh:`1123`)
+- Allow spiders to return dicts. (:gh:`1081`)
+- Add Response.urljoin() helper (:gh:`1086`)
+- look in ~/.config/scrapy.cfg for user config (:gh:`1098`)
+- handle TLS SNI (:gh:`1101`)
+- Selectorlist extract first (:gh:`624`, :gh:`1145`)
+- Added JmesSelect (:gh:`1016`)
+- add gzip compression to filesystem http cache backend (:gh:`1020`)
+- CSS support in link extractors (:gh:`983`)
+- httpcache dont_cache meta #19 #689 (:gh:`821`)
+- add signal to be sent when request is dropped by the scheduler
+ (:gh:`961`)
+- avoid download large response (:gh:`946`)
+- Allow to specify the quotechar in CSVFeedSpider (:gh:`882`)
+- Add referer to "Spider error processing" log message (:gh:`795`)
+- process robots.txt once (:gh:`896`)
+- GSoC Per-spider settings (:gh:`854`)
+- Add project name validation (:gh:`817`)
+- GSoC API cleanup (:gh:`816`, :gh:`1128`, :gh:`1147`,
+ :gh:`1148`, :gh:`1156`, :gh:`1185`, :gh:`1187`, :gh:`1258`,
+ :gh:`1268`, :gh:`1276`, :gh:`1285`, :gh:`1284`)
+- Be more responsive with IO operations (:gh:`1074` and :gh:`1075`)
+- Do leveldb compaction for httpcache on closing (:gh:`1297`)
+
+Deprecations and Removals
+
+- Deprecate htmlparser link extractor (:gh:`1205`)
+- remove deprecated code from FeedExporter (:gh:`1155`)
+- a leftover for.15 compatibility (:gh:`925`)
+- drop support for CONCURRENT_REQUESTS_PER_SPIDER (:gh:`895`)
+- Drop old engine code (:gh:`911`)
+- Deprecate SgmlLinkExtractor (:gh:`777`)
+
+Relocations
+
+- Move exporters/__init__.py to exporters.py (:gh:`1242`)
+- Move base classes to their packages (:gh:`1218`, :gh:`1233`)
+- Module relocation (:gh:`1181`, :gh:`1210`)
+- rename SpiderManager to SpiderLoader (:gh:`1166`)
+- Remove djangoitem (:gh:`1177`)
+- remove scrapy deploy command (:gh:`1102`)
+- dissolve contrib_exp (:gh:`1134`)
+- Deleted bin folder from root, fixes #913 (:gh:`914`)
+- Remove jsonrpc based webservice (:gh:`859`)
+- Move Test cases under project root dir (:gh:`827`, :gh:`841`)
+- Fix backward incompatibility for relocated paths in settings
+ (:gh:`1267`)
+
+Documentation
+
+- CrawlerProcess documentation (:gh:`1190`)
+- Favoring web scraping over screen scraping in the descriptions
+ (:gh:`1188`)
+- Some improvements for Scrapy tutorial (:gh:`1180`)
+- Documenting Files Pipeline together with Images Pipeline (:gh:`1150`)
+- deployment docs tweaks (:gh:`1164`)
+- Added deployment section covering scrapyd-deploy and shub (:gh:`1124`)
+- Adding more settings to project template (:gh:`1073`)
+- some improvements to overview page (:gh:`1106`)
+- Updated link in docs/topics/architecture.rst (:gh:`647`)
+- DOC reorder topics (:gh:`1022`)
+- updating list of Request.meta special keys (:gh:`1071`)
+- DOC document download_timeout (:gh:`898`)
+- DOC simplify extension docs (:gh:`893`)
+- Leaks docs (:gh:`894`)
+- DOC document from_crawler method for item pipelines (:gh:`904`)
+- Spider_error doesn't support deferreds (:gh:`1292`)
+- Corrections & Sphinx related fixes (:gh:`1220`, :gh:`1219`,
+ :gh:`1196`, :gh:`1172`, :gh:`1171`, :gh:`1169`, :gh:`1160`,
+ :gh:`1154`, :gh:`1127`, :gh:`1112`, :gh:`1105`, :gh:`1041`,
+ :gh:`1082`, :gh:`1033`, :gh:`944`, :gh:`866`, :gh:`864`,
+ :gh:`796`, :gh:`1260`, :gh:`1271`, :gh:`1293`, :gh:`1298`)
+
+Bugfixes
+
+- Item multi inheritance fix (:gh:`353`, :gh:`1228`)
+- ItemLoader.load_item: iterate over copy of fields (:gh:`722`)
+- Fix Unhandled error in Deferred (RobotsTxtMiddleware) (:gh:`1131`,
+ :gh:`1197`)
+- Force to read DOWNLOAD_TIMEOUT as int (:gh:`954`)
+- scrapy.utils.misc.load_object should print full traceback (:gh:`902`)
+- Fix bug for ".local" host name (:gh:`878`)
+- Fix for Enabled extensions, middlewares, pipelines info not printed
+ anymore (:gh:`879`)
+- fix dont_merge_cookies bad behaviour when set to false on meta
+ (:gh:`846`)
+
+Python 3 In Progress Support
+
+- disable scrapy.telnet if twisted.conch is not available (:gh:`1161`)
+- fix Python 3 syntax errors in ajaxcrawl.py (:gh:`1162`)
+- more python3 compatibility changes for urllib (:gh:`1121`)
+- assertItemsEqual was renamed to assertCountEqual in Python 3.
+ (:gh:`1070`)
+- Import unittest.mock if available. (:gh:`1066`)
+- updated deprecated cgi.parse_qsl to use six's parse_qsl (:gh:`909`)
+- Prevent Python 3 port regressions (:gh:`830`)
+- PY3: use MutableMapping for python 3 (:gh:`810`)
+- PY3: use six.BytesIO and six.moves.cStringIO (:gh:`803`)
+- PY3: fix xmlrpclib and email imports (:gh:`801`)
+- PY3: use six for robotparser and urlparse (:gh:`800`)
+- PY3: use six.iterkeys, six.iteritems, and tempfile (:gh:`799`)
+- PY3: fix has_key and use six.moves.configparser (:gh:`798`)
+- PY3: use six.moves.cPickle (:gh:`797`)
+- PY3 make it possible to run some tests in Python3 (:gh:`776`)
+
+Tests
+
+- remove unnecessary lines from py3-ignores (:gh:`1243`)
+- Fix remaining warnings from pytest while collecting tests (:gh:`1206`)
+- Add docs build to travis (:gh:`1234`)
+- TST don't collect tests from deprecated modules. (:gh:`1165`)
+- install service_identity package in tests to prevent warnings
+ (:gh:`1168`)
+- Fix deprecated settings API in tests (:gh:`1152`)
+- Add test for webclient with POST method and no body given (:gh:`1089`)
+- py3-ignores.txt supports comments (:gh:`1044`)
+- modernize some of the asserts (:gh:`835`)
+- selector.__repr__ test (:gh:`779`)
+
+Code refactoring
+
+- CSVFeedSpider cleanup: use iterate_spider_output (:gh:`1079`)
+- remove unnecessary check from scrapy.utils.spider.iter_spider_output
+ (:gh:`1078`)
+- Pydispatch pep8 (:gh:`992`)
+- Removed unused 'load=False' parameter from walk_modules() (:gh:`871`)
+- For consistency, use ``job_dir`` helper in ``SpiderState`` extension.
+ (:gh:`805`)
+- rename "sflo" local variables to less cryptic "log_observer" (:gh:`775`)
+
+
+.. _botocore: https://github.com/boto/botocore
+.. _cryptography: https://cryptography.io/en/latest/
+.. _docstrings: https://docs.python.org/3/glossary.html#term-docstring
+.. _KeyboardInterrupt: https://docs.python.org/3/library/exceptions.html#KeyboardInterrupt
+.. _LevelDB: https://github.com/google/leveldb
+.. _lxml: https://lxml.de/
+.. _parsel: https://github.com/scrapy/parsel
+.. _parsel.csstranslator.GenericTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.GenericTranslator
+.. _parsel.csstranslator.HTMLTranslator: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.HTMLTranslator
+.. _parsel.csstranslator.XPathExpr: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.csstranslator.XPathExpr
+.. _PEP 257: https://peps.python.org/pep-0257/
+.. _Pillow: https://github.com/python-pillow/Pillow
+.. _pyOpenSSL: https://www.pyopenssl.org/en/stable/
+.. _queuelib: https://github.com/scrapy/queuelib
+.. _registered with IANA: https://www.iana.org/assignments/media-types/media-types.xhtml
+.. _robots.txt: https://www.robotstxt.org/
+.. _service_identity: https://service-identity.readthedocs.io/en/stable/
+.. _six: https://six.readthedocs.io/
+.. _tox: https://pypi.org/project/tox/
+.. _Twisted: https://twisted.org/
+.. _zope.interface: https://zopeinterface.readthedocs.io/en/latest/
+.. _Zsh: https://www.zsh.org/
diff --git a/docs/news/2.x.rst b/docs/news/2.x.rst
new file mode 100644
index 000000000..5aa62515a
--- /dev/null
+++ b/docs/news/2.x.rst
@@ -0,0 +1,4383 @@
+.. _news-2.x:
+
+Older Scrapy 2.x release notes
+==============================
+
+.. _release-2.13.4:
+
+Scrapy 2.13.4 (2025-11-17)
+--------------------------
+
+Security bug fixes
+~~~~~~~~~~~~~~~~~~
+
+- Improved protection against decompression bombs in
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ for responses compressed using the ``br`` and ``deflate`` methods: if a
+ single compressed chunk would be larger than the response size limit (see
+ :setting:`DOWNLOAD_MAXSIZE`) when decompressed, decompression is no longer
+ carried out. This is especially important for the ``br`` (Brotli) method
+ that can provide a very high compression ratio. Please, see the
+ `CVE-2025-6176`_ and `GHSA-2qfp-q593-8484`_ security advisories for more
+ information.
+ (:gh:`7134`)
+
+ .. _CVE-2025-6176: https://nvd.nist.gov/vuln/detail/CVE-2025-6176
+ .. _GHSA-2qfp-q593-8484: https://github.com/advisories/GHSA-2qfp-q593-8484
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- The minimum supported version of the optional ``brotli`` package is now
+ ``1.2.0``.
+ (:gh:`7134`)
+
+- The ``brotlicffi`` and ``brotlipy`` packages can no longer be used to
+ decompress Brotli-compressed responses. Please install the ``brotli``
+ package instead.
+ (:gh:`7134`)
+
+Other changes
+~~~~~~~~~~~~~
+
+- Restricted the maximum supported Twisted version to ``25.5.0``, as Scrapy
+ currently uses some private APIs changed in later Twisted versions.
+ (:gh:`7142`)
+
+- Stopped setting the ``COVERAGE_CORE`` environment variable in tests, it
+ didn't have an effect but caused the ``coverage`` module to produce a
+ warning or an error.
+ (:gh:`7137`)
+
+- Removed the documentation build dependency on the deprecated
+ ``sphinx-hoverxref`` module.
+ (:gh:`6786`, :gh:`6922`)
+
+.. _release-2.13.3:
+
+Scrapy 2.13.3 (2025-07-02)
+--------------------------
+
+- Changed the values for :setting:`DOWNLOAD_DELAY` (from ``0`` to ``1``) and
+ :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (from ``8`` to ``1``) in the
+ default project template.
+ (:gh:`6597`, :gh:`6918`, :gh:`6923`)
+
+- Improved :class:`scrapy.core.engine.ExecutionEngine` logic related to
+ initialization and exception handling, fixing several cases where the
+ spider would crash, hang or log an unhandled exception.
+ (:gh:`6783`, :gh:`6784`, :gh:`6900`, :gh:`6908`, :gh:`6910`,
+ :gh:`6911`)
+
+- Fixed a Windows issue with :ref:`feed exports <topics-feed-exports>` using
+ :class:`scrapy.extensions.feedexport.FileFeedStorage` that caused the file
+ to be created on the wrong drive.
+ (:gh:`6894`, :gh:`6897`)
+
+- Allowed running tests with Twisted 25.5.0+ again. Pytest 8.4.1+ is now
+ required for running tests in non-pinned envs as support for the new
+ Twisted version was added in that version.
+ (:gh:`6893`)
+
+- Fixed running tests with lxml 6.0.0+.
+ (:gh:`6919`)
+
+- Added a deprecation notice for
+ ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` to :ref:`the Scrapy
+ 2.11.2 release notes <release-2.11.2>`.
+ (:gh:`6926`)
+
+- Updated :ref:`contribution docs <topics-contributing>` to refer to ruff_
+ instead of black_.
+ (:gh:`6903`)
+
+- Added ``.venv/`` and ``.vscode/`` to ``.gitignore``.
+ (:gh:`6901`, :gh:`6907`)
+
+
+.. _release-2.13.2:
+
+Scrapy 2.13.2 (2025-06-09)
+--------------------------
+
+- Fixed a bug introduced in Scrapy 2.13.0 that caused results of request
+ errbacks to be ignored when the errback was called because of a downloader
+ error.
+ (:gh:`6861`, :gh:`6863`)
+
+- Added a note about the behavior change of
+ ``scrapy.utils.reactor.is_asyncio_reactor_installed`` to its docs and
+ to the "Backward-incompatible changes" section of :ref:`the Scrapy 2.13.0
+ release notes <release-2.13.0>`.
+ (:gh:`6866`)
+
+- Improved the message in the exception raised by
+ :func:`scrapy.utils.test.get_reactor_settings` when there is no reactor
+ installed.
+ (:gh:`6866`)
+
+- Updated the :class:`scrapy.crawler.CrawlerRunner` examples in
+ :ref:`topics-practices` to install the reactor explicitly, to fix
+ reactor-related errors with Scrapy 2.13.0 and later.
+ (:gh:`6865`)
+
+- Fixed ``scrapy fetch`` not working with scrapy-poet_.
+ (:gh:`6872`)
+
+- Fixed an exception produced by :class:`scrapy.core.engine.ExecutionEngine`
+ when it's closed before being fully initialized.
+ (:gh:`6857`, :gh:`6867`)
+
+- Improved the README, updated the Scrapy logo in it.
+ (:gh:`6831`, :gh:`6833`, :gh:`6839`)
+
+- Restricted the Twisted version used in tests to below 25.5.0, as some tests
+ fail with 25.5.0.
+ (:gh:`6878`, :gh:`6882`)
+
+- Updated type hints for Twisted 25.5.0 changes.
+ (:gh:`6882`)
+
+- Removed the old artwork.
+ (:gh:`6874`)
+
+
+.. _release-2.13.1:
+
+Scrapy 2.13.1 (2025-05-28)
+--------------------------
+
+- Give callback requests precedence over start requests when priority values
+ are the same.
+
+ This makes changes from 2.13.0 to start request handling more intuitive and
+ backward compatible. For scenarios where all requests have the same
+ priorities, in 2.13.0 all start requests were sent before the first
+ callback request. In 2.13.1, same as in 2.12 and lower, start requests are
+ only sent when there are not enough pending callback requests to reach
+ concurrency limits.
+
+ (:gh:`6828`)
+
+- Added a deepwiki_ badge to the README. (:gh:`6793`)
+
+ .. _deepwiki: https://deepwiki.com/scrapy/scrapy
+
+- Fixed a typo in the code example of :ref:`start-requests-lazy`.
+ (:gh:`6812`, :gh:`6815`)
+
+- Fixed a typo in the :ref:`coroutine-support` section of the documentation.
+ (:gh:`6822`)
+
+- Made this page more prominently listed in PyPI project links.
+ (:gh:`6826`)
+
+
+.. _release-2.13.0:
+
+Scrapy 2.13.0 (2025-05-08)
+--------------------------
+
+Highlights:
+
+- The asyncio reactor is now enabled by default
+
+- Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`
+ (async) and changed how it is iterated
+
+- Added the :reqmeta:`allow_offsite` request meta key
+
+- Spider middlewares that don't support asynchronous spider output are
+ deprecated
+
+- Added a base class for :ref:`universal spider middlewares
+ <universal-spider-middleware>`
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Dropped support for PyPy 3.9.
+ (:gh:`6613`)
+
+- Added support for PyPy 3.11.
+ (:gh:`6697`)
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- The default value of the :setting:`TWISTED_REACTOR` setting was changed
+ from ``None`` to
+ ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``. This value
+ was used in newly generated projects since Scrapy 2.7.0 but now existing
+ projects that don't explicitly set this setting will also use the asyncio
+ reactor. You can :ref:`change this setting in your project
+ <disable-asyncio>` to use a different reactor.
+ (:gh:`6659`, :gh:`6713`)
+
+- The iteration of start requests and items no longer stops once there are
+ requests in the scheduler, and instead runs continuously until all start
+ requests have been scheduled.
+
+ To reproduce the previous behavior, see :ref:`start-requests-lazy`.
+ (:gh:`6729`)
+
+- An unhandled exception from the
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.open_spider` method of a
+ :ref:`spider middleware <topics-spider-middleware>` no longer stops the
+ crawl.
+ (:gh:`6729`)
+
+- In ``scrapy.core.engine.ExecutionEngine``:
+
+ - The second parameter of ``open_spider()``, ``start_requests``, has been
+ removed. The start requests are determined by the ``spider`` parameter
+ instead (see :meth:`~scrapy.Spider.start`).
+
+ - The ``slot`` attribute has been renamed to ``_slot`` and should not be
+ used.
+
+ (:gh:`6729`)
+
+- In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot``
+ and should not be used.
+ (:gh:`6729`)
+
+- The ``slot`` :ref:`telnet variable <telnet-vars>` has been removed.
+ (:gh:`6729`)
+
+- In ``scrapy.core.spidermw.SpiderMiddlewareManager``,
+ ``process_start_requests()`` has been replaced by ``process_start()``.
+ (:gh:`6729`)
+
+- The ``scrape_func`` callable passed to
+ ``scrapy.core.spidermw.SpiderMiddlewareManager.scrape_response()`` is now
+ called with 2 parameters, ``response`` and ``request``, instead of 3, and
+ must return a :class:`~twisted.internet.defer.Deferred` instead of an
+ iterable.
+ (:gh:`6787`)
+
+- The now-deprecated ``start_requests()`` method, when it returns an iterable
+ instead of being defined as a generator, is now executed *after* the
+ :ref:`scheduler <topics-scheduler>` instance has been created.
+ (:gh:`6729`)
+
+- When using :setting:`JOBDIR`, :ref:`start requests <start-requests>` are
+ now serialized into their own, ``s``-suffixed priority folders. You can set
+ :setting:`SCHEDULER_START_DISK_QUEUE` to ``None`` or ``""`` to change that,
+ but the side effects may be undesirable. See
+ :setting:`SCHEDULER_START_DISK_QUEUE` for details.
+ (:gh:`6729`)
+
+- The URL length limit, set by the :setting:`URLLENGTH_LIMIT` setting, is now
+ also enforced for start requests.
+ (:gh:`6777`)
+
+- Calling ``scrapy.utils.reactor.is_asyncio_reactor_installed`` without
+ an installed reactor now raises an exception instead of installing a
+ reactor. This shouldn't affect normal Scrapy use cases, but it may affect
+ 3rd-party test suites that use Scrapy internals such as
+ :class:`~scrapy.crawler.Crawler` and don't install a reactor explicitly. If
+ you are affected by this change, you most likely need to install the
+ reactor before running Scrapy code that expects it to be installed.
+ (:gh:`6732`, :gh:`6735`)
+
+- The ``from_settings()`` method of
+ :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`,
+ deprecated in Scrapy 2.12.0, is removed earlier than the usual deprecation
+ period (this was needed because after the introduction of the
+ :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware` base class and
+ switching built-in spider middlewares to it those middlewares need the
+ :class:`~scrapy.crawler.Crawler` instance at run time). Please use
+ ``from_crawler()`` instead.
+ (:gh:`6693`)
+
+- ``scrapy.utils.url.escape_ajax()`` is no longer called when a
+ :class:`~scrapy.Request` instance is created. It was only useful for
+ websites supporting the ``_escaped_fragment_`` feature which most modern
+ websites don't support. If you still need this you can modify the URLs
+ before passing them to :class:`~scrapy.Request`.
+ (:gh:`6523`, :gh:`6651`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- Removed old deprecated name aliases for some signals:
+
+ - ``stats_spider_opened`` (use ``spider_opened`` instead)
+
+ - ``stats_spider_closing`` and ``stats_spider_closed`` (use
+ ``spider_closed`` instead)
+
+ - ``item_passed`` (use ``item_scraped`` instead)
+
+ - ``request_received`` (use ``request_scheduled`` instead)
+
+ (:gh:`6654`, :gh:`6655`)
+
+Deprecations
+~~~~~~~~~~~~
+
+- The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated,
+ use :meth:`~scrapy.Spider.start` instead, or both to maintain support for
+ lower Scrapy versions.
+ (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
+
+- The ``process_start_requests()`` method of :ref:`spider middlewares
+ <topics-spider-middleware>` is deprecated, use
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
+ or both to maintain support for lower Scrapy versions.
+ (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
+
+- The ``__init__`` method of priority queue classes (see
+ :setting:`SCHEDULER_PRIORITY_QUEUE`) should now support a keyword-only
+ ``start_queue_cls`` parameter.
+ (:gh:`6752`)
+
+- Spider middlewares that don't support asynchronous spider output are
+ deprecated. The async iterable downgrading feature, needed for using such
+ middlewares with asynchronous callbacks and with other spider middlewares
+ that produce asynchronous iterables, is also deprecated. Please update all
+ such middlewares to support asynchronous spider output. (:gh:`6664`)
+
+- Functions that were imported from :mod:`w3lib.url` and re-exported in
+ :mod:`scrapy.utils.url` are now deprecated, you should import them from
+ :mod:`w3lib.url` directly. They are:
+
+ - ``scrapy.utils.url.add_or_replace_parameter()``
+
+ - ``scrapy.utils.url.add_or_replace_parameters()``
+
+ - ``scrapy.utils.url.any_to_uri()``
+
+ - ``scrapy.utils.url.canonicalize_url()``
+
+ - ``scrapy.utils.url.file_uri_to_path()``
+
+ - ``scrapy.utils.url.is_url()``
+
+ - ``scrapy.utils.url.parse_data_uri()``
+
+ - ``scrapy.utils.url.parse_url()``
+
+ - ``scrapy.utils.url.path_to_file_uri()``
+
+ - ``scrapy.utils.url.safe_download_url()``
+
+ - ``scrapy.utils.url.safe_url_string()``
+
+ - ``scrapy.utils.url.url_query_cleaner()``
+
+ - ``scrapy.utils.url.url_query_parameter()``
+
+ (:gh:`4577`, :gh:`6583`, :gh:`6586`)
+
+- HTTP/1.0 support code is deprecated. It was disabled by default and
+ couldn't be used together with HTTP/1.1. If you still need it, you should
+ write your own download handler or copy the code from Scrapy. The
+ deprecations include:
+
+ - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPClientFactory``
+
+ - ``scrapy.core.downloader.webclient.ScrapyHTTPPageGetter``
+
+ - Overriding
+ ``scrapy.core.downloader.contextfactory.ScrapyClientContextFactory.getContext()``
+
+ (:gh:`6634`)
+
+- The following modules and functions used only in tests are deprecated:
+
+ - the ``scrapy.utils.testproc`` module
+
+ - the ``scrapy.utils.testsite`` module
+
+ - ``scrapy.utils.test.assert_gcs_environ()``
+
+ - ``scrapy.utils.test.get_ftp_content_and_delete()``
+
+ - ``scrapy.utils.test.get_gcs_content_and_delete()``
+
+ - ``scrapy.utils.test.mock_google_cloud_storage()``
+
+ - ``scrapy.utils.test.skip_if_no_boto()``
+
+ If you need to use them in your tests or code, you can copy the code from Scrapy.
+ (:gh:`6696`)
+
+- ``scrapy.utils.test.TestSpider`` is deprecated. If you need an empty spider
+ class you can use :class:`scrapy.utils.spider.DefaultSpider` or create your
+ own subclass of :class:`scrapy.Spider`.
+ (:gh:`6678`)
+
+- ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` is
+ deprecated. It was disabled by default and isn't useful for most of the
+ existing websites.
+ (:gh:`6523`, :gh:`6651`, :gh:`6656`)
+
+- ``scrapy.utils.url.escape_ajax()`` is deprecated.
+ (:gh:`6523`, :gh:`6651`)
+
+- ``scrapy.spiders.init.InitSpider`` is deprecated. If you find it useful,
+ you can copy its code from Scrapy.
+ (:gh:`6708`, :gh:`6714`)
+
+- ``scrapy.utils.versions.scrapy_components_versions()`` is deprecated, use
+ ``scrapy.utils.versions.get_versions()`` instead.
+ (:gh:`6582`)
+
+- ``BaseDupeFilter.log()`` is deprecated. It does nothing and shouldn't be
+ called.
+ (:gh:`4151`)
+
+- Passing the ``spider`` argument to the following methods of
+ :class:`~scrapy.core.scraper.Scraper` is deprecated:
+
+ - ``close_spider()``
+
+ - ``enqueue_scrape()``
+
+ - ``handle_spider_error()``
+
+ - ``handle_spider_output()``
+
+ (:gh:`6764`)
+
+New features
+~~~~~~~~~~~~
+
+- You can now yield the start requests and items of a spider from the
+ :meth:`~scrapy.Spider.start` spider method and from the
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` spider
+ middleware method, both :term:`asynchronous generators <python:asynchronous
+ generator>`.
+
+ This makes it possible to use asynchronous code to generate those start
+ requests and items, e.g. reading them from a queue service or database
+ using an asynchronous client, without workarounds.
+ (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
+
+- Start requests are now :ref:`scheduled <topics-scheduler>` as soon as
+ possible.
+
+ As a result, their :attr:`~scrapy.Request.priority` is now taken into
+ account as soon as :setting:`CONCURRENT_REQUESTS` is reached.
+ (:gh:`456`, :gh:`3477`, :gh:`4467`, :gh:`5627`, :gh:`6729`)
+
+- :class:`Crawler.signals <scrapy.signalmanager.SignalManager>` has a new
+ :meth:`~scrapy.signalmanager.SignalManager.wait_for` method.
+ (:gh:`6729`)
+
+- Added a new :signal:`scheduler_empty` signal.
+ (:gh:`6729`)
+
+- Added new settings: :setting:`SCHEDULER_START_DISK_QUEUE` and
+ :setting:`SCHEDULER_START_MEMORY_QUEUE`.
+ (:gh:`6729`)
+
+- Added :class:`~scrapy.spidermiddlewares.start.StartSpiderMiddleware`, which
+ sets :reqmeta:`is_start_request` to ``True`` on :ref:`start requests
+ <start-requests>`.
+ (:gh:`6729`)
+
+- Exposed a new method of :class:`Crawler.engine
+ <scrapy.core.engine.ExecutionEngine>`:
+ :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`.
+ (:gh:`6729`)
+
+- Added the :reqmeta:`allow_offsite` request meta key that can be used
+ instead of the more general :attr:`~scrapy.Request.dont_filter` request
+ attribute to skip processing of the request by
+ :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` (but not
+ by other code that checks :attr:`~scrapy.Request.dont_filter`).
+ (:gh:`3690`, :gh:`6151`, :gh:`6366`)
+
+- Added an optional base class for spider middlewares,
+ :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which can be
+ helpful for writing :ref:`universal spider middlewares
+ <universal-spider-middleware>` without boilerplate and code duplication.
+ The built-in spider middlewares now inherit from this class.
+ (:gh:`6693`, :gh:`6777`)
+
+- :ref:`Scrapy add-ons <topics-addons>` can now define a class method called
+ ``update_pre_crawler_settings()`` to update :ref:`pre-crawler settings
+ <pre-crawler-settings>`.
+ (:gh:`6544`, :gh:`6568`)
+
+- Added :ref:`helpers <priority-dict-helpers>` for modifying :ref:`component
+ priority dictionary <component-priority-dictionaries>` settings.
+ (:gh:`6614`)
+
+- Responses that use an unknown/unsupported encoding now produce a warning.
+ If Scrapy knows that installing an additional package (such as brotli_)
+ will allow decoding the response, that will be mentioned in the warning.
+ (:gh:`4697`, :gh:`6618`)
+
+- Added the ``spider_exceptions/count`` stat which tracks the total count of
+ exceptions (tracked also by per-type ``spider_exceptions/*`` stats).
+ (:gh:`6739`, :gh:`6740`)
+
+- Added the :setting:`DEFAULT_DROPITEM_LOG_LEVEL` setting and the
+ :attr:`scrapy.exceptions.DropItem.log_level` attribute that allow
+ customizing the log level of the message that is logged when an item is
+ dropped.
+ (:gh:`6603`, :gh:`6608`)
+
+- Added support for the ``-b, --cookie`` curl argument to
+ :meth:`scrapy.Request.from_curl`.
+ (:gh:`6684`)
+
+- Added the :setting:`LOG_VERSIONS` setting that allows customizing the
+ list of software whose versions are logged when the spider starts.
+ (:gh:`6582`)
+
+- Added the :setting:`WARN_ON_GENERATOR_RETURN_VALUE` setting that allows
+ disabling run time analysis of callback code used to warn about incorrect
+ ``return`` statements in generator-based callbacks. You may need to disable
+ this setting if this analysis breaks on your callback code.
+ (:gh:`6731`, :gh:`6738`)
+
+Improvements
+~~~~~~~~~~~~
+
+- Removed or postponed some calls of :func:`itemadapter.is_item` to increase
+ performance.
+ (:gh:`6719`)
+
+- Improved the error message when running a ``scrapy`` command that requires
+ a project (such as ``scrapy crawl``) outside of a project directory.
+ (:gh:`2349`, :gh:`3426`)
+
+- Added an empty :setting:`ADDONS` setting to the ``settings.py`` template
+ for new projects.
+ (:gh:`6587`)
+
+Bug fixes
+~~~~~~~~~
+
+- Yielding an item from :meth:`Spider.start <scrapy.Spider.start>` or from
+ :meth:`SpiderMiddleware.process_start
+ <scrapy.spidermiddlewares.SpiderMiddleware.process_start>` no longer delays
+ the next iteration of starting requests and items by up to 5 seconds.
+ (:gh:`6729`)
+
+- Fixed calculation of ``items_per_minute`` and ``responses_per_minute``
+ stats.
+ (:gh:`6599`)
+
+- Fixed an error initializing
+ :class:`scrapy.extensions.feedexport.GCSFeedStorage`.
+ (:gh:`6617`, :gh:`6628`)
+
+- Fixed an error running ``scrapy bench``.
+ (:gh:`6632`, :gh:`6633`)
+
+- Fixed duplicated log messages about the reactor and the event loop.
+ (:gh:`6636`, :gh:`6657`)
+
+- Fixed resolving type annotations of ``SitemapSpider._parse_sitemap()`` at
+ run time, required by tools such as scrapy-poet_.
+ (:gh:`6665`, :gh:`6671`)
+
+ .. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet
+
+- Calling ``scrapy.utils.reactor.is_asyncio_reactor_installed`` without
+ an installed reactor now raises an exception instead of installing a
+ reactor.
+ (:gh:`6732`, :gh:`6735`)
+
+- Restored support for the ``x-gzip`` content encoding.
+ (:gh:`6618`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Documented the setting values set in the default project template.
+ (:gh:`6762`, :gh:`6775`)
+
+- Improved the docs about asynchronous iterable support in spider
+ middlewares. (:gh:`6688`)
+
+- Improved the :ref:`docs <coroutine-deferred-apis>` about using
+ :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based
+ code and included a list of such APIs.
+ (:gh:`6677`, :gh:`6734`, :gh:`6776`)
+
+- Improved the :ref:`contribution docs <topics-contributing>`.
+ (:gh:`6561`, :gh:`6575`)
+
+- Removed the ``Splash`` recommendation from the :ref:`headless browser
+ <topics-headless-browsing>` suggestion. We no longer recommend using
+ ``Splash`` and recommend using other headless browser solutions instead.
+ (:gh:`6642`, :gh:`6701`)
+
+- Added the dark mode to the HTML documentation.
+ (:gh:`6653`)
+
+- Other documentation improvements and fixes.
+ (:gh:`4151`,
+ :gh:`6526`,
+ :gh:`6620`,
+ :gh:`6621`,
+ :gh:`6622`,
+ :gh:`6623`,
+ :gh:`6624`,
+ :gh:`6721`,
+ :gh:`6723`,
+ :gh:`6780`)
+
+Packaging
+~~~~~~~~~
+
+- Switched from ``setup.py`` to ``pyproject.toml``.
+ (:gh:`6514`, :gh:`6547`)
+
+- Switched the build backend from setuptools_ to hatchling_.
+ (:gh:`6771`)
+
+ .. _hatchling: https://pypi.org/project/hatchling/
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Replaced most linters with ruff_.
+ (:gh:`6565`,
+ :gh:`6576`,
+ :gh:`6577`,
+ :gh:`6581`,
+ :gh:`6584`,
+ :gh:`6595`,
+ :gh:`6601`,
+ :gh:`6631`)
+
+ .. _ruff: https://docs.astral.sh/ruff/
+
+- Improved accuracy and performance of collecting test coverage.
+ (:gh:`6255`, :gh:`6610`)
+
+- Fixed an error that prevented running tests from directories other than the
+ top level source directory.
+ (:gh:`6567`)
+
+- Reduced the amount of ``mockserver`` calls in tests to improve the overall
+ test run time.
+ (:gh:`6637`, :gh:`6648`)
+
+- Fixed tests that were running the same test code more than once.
+ (:gh:`6646`, :gh:`6647`, :gh:`6650`)
+
+- Refactored tests to use more ``pytest`` features instead of ``unittest``
+ ones where possible.
+ (:gh:`6678`,
+ :gh:`6680`,
+ :gh:`6695`,
+ :gh:`6699`,
+ :gh:`6700`,
+ :gh:`6702`,
+ :gh:`6709`,
+ :gh:`6710`,
+ :gh:`6711`,
+ :gh:`6712`,
+ :gh:`6725`)
+
+- Type hints improvements and fixes.
+ (:gh:`6578`,
+ :gh:`6579`,
+ :gh:`6593`,
+ :gh:`6605`,
+ :gh:`6694`)
+
+- CI and test improvements and fixes.
+ (:gh:`5360`,
+ :gh:`6271`,
+ :gh:`6547`,
+ :gh:`6560`,
+ :gh:`6602`,
+ :gh:`6607`,
+ :gh:`6609`,
+ :gh:`6613`,
+ :gh:`6619`,
+ :gh:`6626`,
+ :gh:`6679`,
+ :gh:`6703`,
+ :gh:`6704`,
+ :gh:`6716`,
+ :gh:`6720`,
+ :gh:`6722`,
+ :gh:`6724`,
+ :gh:`6741`,
+ :gh:`6743`,
+ :gh:`6766`,
+ :gh:`6770`,
+ :gh:`6772`,
+ :gh:`6773`)
+
+- Code cleanups.
+ (:gh:`6600`,
+ :gh:`6606`,
+ :gh:`6635`,
+ :gh:`6764`)
+
+
+.. _release-2.12.0:
+
+Scrapy 2.12.0 (2024-11-18)
+--------------------------
+
+Highlights:
+
+- Dropped support for Python 3.8, added support for Python 3.13
+
+- ``scrapy.Spider.start_requests()`` can now yield items
+
+- Added :class:`~scrapy.http.JsonResponse`
+
+- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Dropped support for Python 3.8.
+ (:gh:`6466`, :gh:`6472`)
+
+- Added support for Python 3.13.
+ (:gh:`6166`)
+
+- Minimum versions increased for these dependencies:
+
+ - Twisted_: 18.9.0 → 21.7.0
+
+ - cryptography_: 36.0.0 → 37.0.0
+
+ - pyOpenSSL_: 21.0.0 → 22.0.0
+
+ - lxml_: 4.4.1 → 4.6.0
+
+- Removed ``setuptools`` from the dependency list.
+ (:gh:`6487`)
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- User-defined cookies for HTTPS requests will have the ``secure`` flag set
+ to ``True`` unless it's set to ``False`` explicitly. This is important when
+ these cookies are reused in HTTP requests, e.g. after a redirect to an HTTP
+ URL.
+ (:gh:`6357`)
+
+- The Reppy-based ``robots.txt`` parser,
+ ``scrapy.robotstxt.ReppyRobotParser``, was removed, as it doesn't support
+ Python 3.9+.
+ (:gh:`5230`, :gh:`6099`, :gh:`6499`)
+
+- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
+ its subclasses was improved and it's possible that some previously working
+ usage scenarios will no longer work. It can only affect you if you define
+ custom subclasses of ``MediaPipeline`` or create instances of these
+ pipelines via ``from_settings()`` or ``__init__()`` calls instead of
+ ``from_crawler()`` calls.
+
+ Previously, ``MediaPipeline.from_crawler()`` called the ``from_settings()``
+ method if it existed or the ``__init__()`` method otherwise, and then did
+ some additional initialization using the ``crawler`` instance. If the
+ ``from_settings()`` method existed (like in ``FilesPipeline``) it called
+ ``__init__()`` to create the instance. It wasn't possible to override
+ ``from_crawler()`` without calling ``MediaPipeline.from_crawler()`` from it
+ which, in turn, couldn't be called in some cases (including subclasses of
+ ``FilesPipeline``).
+
+ Now, in line with the general usage of ``from_crawler()`` and
+ ``from_settings()`` and the deprecation of the latter the recommended
+ initialization order is the following one:
+
+ - All ``__init__()`` methods should take a ``crawler`` argument. If they
+ also take a ``settings`` argument they should ignore it, using
+ ``crawler.settings`` instead. When they call ``__init__()`` of the base
+ class they should pass the ``crawler`` argument to it too.
+ - A ``from_settings()`` method shouldn't be defined. Class-specific
+ initialization code should go into either an overridden ``from_crawler()``
+ method or into ``__init__()``.
+ - It's now possible to override ``from_crawler()`` and it's not necessary
+ to call ``MediaPipeline.from_crawler()`` in it if other recommendations
+ were followed.
+ - If pipeline instances were created with ``from_settings()`` or
+ ``__init__()`` calls (which wasn't supported even before, as it missed
+ important initialization code), they should now be created with
+ ``from_crawler()`` calls.
+
+ (:gh:`6540`)
+
+- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
+ <scrapy.pipelines.images.ImagesPipeline.convert_image>` is now
+ positional-only, as it was changed from optional to required.
+ (:gh:`6500`)
+
+- The ``convert`` argument of :func:`scrapy.utils.conf.build_component_list`
+ is now positional-only, as the preceding argument (``custom``) was removed.
+ (:gh:`6500`)
+
+- The ``overwrite_output`` argument of
+ :func:`scrapy.utils.conf.feed_process_params_from_cli` is now
+ positional-only, as the preceding argument (``output_format``) was removed.
+ (:gh:`6500`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- Removed the ``scrapy.utils.request.request_fingerprint()`` function,
+ deprecated in Scrapy 2.7.0.
+ (:gh:`6212`, :gh:`6213`)
+
+- Removed support for value ``"2.6"`` of setting
+ ``REQUEST_FINGERPRINTER_IMPLEMENTATION``, deprecated in Scrapy 2.7.0.
+ (:gh:`6212`, :gh:`6213`)
+
+- :class:`~scrapy.dupefilters.RFPDupeFilter` subclasses now require
+ supporting the ``fingerprinter`` parameter in their ``__init__`` method,
+ introduced in Scrapy 2.7.0.
+ (:gh:`6102`, :gh:`6113`)
+
+- Removed the ``scrapy.downloadermiddlewares.decompression`` module,
+ deprecated in Scrapy 2.7.0.
+ (:gh:`6100`, :gh:`6113`)
+
+- Removed the ``scrapy.utils.response.response_httprepr()`` function,
+ deprecated in Scrapy 2.6.0.
+ (:gh:`6111`, :gh:`6116`)
+
+- Spiders with spider-level HTTP authentication, i.e. with the ``http_user``
+ or ``http_pass`` attributes, must now define ``http_auth_domain`` as well,
+ which was introduced in Scrapy 2.5.1.
+ (:gh:`6103`, :gh:`6113`)
+
+- :ref:`Media pipelines <topics-media-pipeline>` methods ``file_path()``,
+ ``file_downloaded()``, ``get_images()``, ``image_downloaded()``,
+ ``media_downloaded()``, ``media_to_download()``, and ``thumb_path()`` must
+ now support an ``item`` parameter, added in Scrapy 2.4.0.
+ (:gh:`6107`, :gh:`6113`)
+
+- The ``__init__()`` and ``from_crawler()`` methods of :ref:`feed storage
+ backend classes <topics-feed-storage>` must now support the keyword-only
+ ``feed_options`` parameter, introduced in Scrapy 2.4.0.
+ (:gh:`6105`, :gh:`6113`)
+
+- Removed the ``scrapy.loader.common`` and ``scrapy.loader.processors``
+ modules, deprecated in Scrapy 2.3.0.
+ (:gh:`6106`, :gh:`6113`)
+
+- Removed the ``scrapy.utils.misc.extract_regex()`` function, deprecated in
+ Scrapy 2.3.0.
+ (:gh:`6106`, :gh:`6113`)
+
+- Removed the ``scrapy.http.JSONRequest`` class, replaced with
+ ``JsonRequest`` in Scrapy 1.8.0.
+ (:gh:`6110`, :gh:`6113`)
+
+- ``scrapy.utils.log.logformatter_adapter`` no longer supports missing
+ ``args``, ``level``, or ``msg`` parameters, and no longer supports a
+ ``format`` parameter, all scenarios that were deprecated in Scrapy 1.0.0.
+ (:gh:`6109`, :gh:`6116`)
+
+- A custom class assigned to the :setting:`SPIDER_LOADER_CLASS` setting that
+ does not implement the ``ISpiderLoader`` interface
+ will now raise a :exc:`zope.interface.verify.DoesNotImplement` exception at
+ run time. Non-compliant classes have been triggering a deprecation warning
+ since Scrapy 1.0.0.
+ (:gh:`6101`, :gh:`6113`)
+
+- Removed the ``--output-format``/``-t`` command line option, deprecated in
+ Scrapy 2.1.0. ``-O <URI>:<FORMAT>`` should be used instead.
+ (:gh:`6500`)
+
+- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
+ :class:`~scrapy.crawler.Crawler` instance, deprecated in Scrapy 2.11.0, now
+ raises an exception.
+ (:gh:`6500`)
+
+- Subclassing
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ without support for the ``crawler`` argument in ``__init__()`` and without
+ a custom ``from_crawler()`` method, deprecated in Scrapy 2.5.0, is no
+ longer allowed.
+ (:gh:`6500`)
+
+- Removed the ``EXCEPTIONS_TO_RETRY`` attribute of
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`, deprecated in
+ Scrapy 2.10.0.
+ (:gh:`6500`)
+
+- Removed support for :ref:`S3 feed exports <topics-feed-storage-s3>` without
+ the boto3_ package installed, deprecated in Scrapy 2.10.0.
+ (:gh:`6500`)
+
+- Removed the ``scrapy.extensions.feedexport._FeedSlot`` class, deprecated in
+ Scrapy 2.10.0.
+ (:gh:`6500`)
+
+- Removed the ``scrapy.pipelines.images.NoimagesDrop`` exception, deprecated
+ in Scrapy 2.8.0.
+ (:gh:`6500`)
+
+- The ``response_body`` argument of :meth:`ImagesPipeline.convert_image
+ <scrapy.pipelines.images.ImagesPipeline.convert_image>` is now required,
+ not passing it was deprecated in Scrapy 2.8.0.
+ (:gh:`6500`)
+
+- Removed the ``custom`` argument of
+ :func:`scrapy.utils.conf.build_component_list`, deprecated in Scrapy
+ 2.10.0.
+ (:gh:`6500`)
+
+- Removed the ``scrapy.utils.reactor.get_asyncio_event_loop_policy()``
+ function, deprecated in Scrapy 2.9.0. Use :func:`asyncio.get_event_loop`
+ and related standard library functions instead.
+ (:gh:`6500`)
+
+Deprecations
+~~~~~~~~~~~~
+
+- The ``from_settings()`` methods of the :ref:`Scrapy components
+ <topics-components>` that have them are now deprecated. ``from_crawler()``
+ should now be used instead. Affected components:
+
+ - :class:`scrapy.dupefilters.RFPDupeFilter`
+ - :class:`scrapy.mail.MailSender`
+ - :class:`scrapy.middleware.MiddlewareManager`
+ - :class:`scrapy.core.downloader.contextfactory.ScrapyClientContextFactory`
+ - :class:`scrapy.pipelines.files.FilesPipeline`
+ - :class:`scrapy.pipelines.images.ImagesPipeline`
+ - :class:`scrapy.spidermiddlewares.urllength.UrlLengthMiddleware`
+
+ (:gh:`6540`)
+
+- It's now deprecated to have a ``from_settings()`` method but no
+ ``from_crawler()`` method in 3rd-party :ref:`Scrapy components
+ <topics-components>`. You can define a simple ``from_crawler()`` method
+ that calls ``cls.from_settings(crawler.settings)`` to fix this if you don't
+ want to refactor the code. Note that if you have a ``from_crawler()``
+ method Scrapy will not call the ``from_settings()`` method so the latter
+ can be removed.
+ (:gh:`6540`)
+
+- The initialization API of :class:`scrapy.pipelines.media.MediaPipeline` and
+ its subclasses was improved and some old usage scenarios are now deprecated
+ (see also the "Backward-incompatible changes" section). Specifically:
+
+ - It's deprecated to define an ``__init__()`` method that doesn't take a
+ ``crawler`` argument.
+ - It's deprecated to call an ``__init__()`` method without passing a
+ ``crawler`` argument. If it's passed, it's also deprecated to pass a
+ ``settings`` argument, which will be ignored anyway.
+ - Calling ``from_settings()`` is deprecated, use ``from_crawler()``
+ instead.
+ - Overriding ``from_settings()`` is deprecated, override ``from_crawler()``
+ instead.
+
+ (:gh:`6540`)
+
+- The ``REQUEST_FINGERPRINTER_IMPLEMENTATION`` setting is now deprecated.
+ (:gh:`6212`, :gh:`6213`)
+
+- The ``scrapy.utils.misc.create_instance()`` function is now deprecated, use
+ :func:`scrapy.utils.misc.build_from_crawler` instead.
+ (:gh:`5523`, :gh:`5884`, :gh:`6162`, :gh:`6169`, :gh:`6540`)
+
+- ``scrapy.core.downloader.Downloader._get_slot_key()`` is deprecated, use
+ :meth:`scrapy.core.downloader.Downloader.get_slot_key` instead.
+ (:gh:`6340`, :gh:`6352`)
+
+- ``scrapy.utils.defer.process_chain_both()`` is now deprecated.
+ (:gh:`6397`)
+
+- ``scrapy.twisted_version`` is now deprecated, you should instead use
+ :attr:`twisted.version` directly (but note that it's an
+ ``incremental.Version`` object, not a tuple).
+ (:gh:`6509`, :gh:`6512`)
+
+- ``scrapy.utils.python.flatten()`` and ``scrapy.utils.python.iflatten()``
+ are now deprecated.
+ (:gh:`6517`, :gh:`6519`)
+
+- ``scrapy.utils.python.equal_attributes()`` is now deprecated.
+ (:gh:`6517`, :gh:`6519`)
+
+- ``scrapy.utils.request.request_authenticate()`` is now deprecated, you
+ should instead just set the ``Authorization`` header directly.
+ (:gh:`6517`, :gh:`6519`)
+
+- ``scrapy.utils.serialize.ScrapyJSONDecoder`` is now deprecated, it didn't
+ contain any code since Scrapy 1.0.0.
+ (:gh:`6517`, :gh:`6519`)
+
+- ``scrapy.utils.test.assert_samelines()`` is now deprecated.
+ (:gh:`6517`, :gh:`6519`)
+
+- ``scrapy.extensions.feedexport.build_storage()`` is now deprecated. You can
+ instead call the builder callable directly.
+ (:gh:`6540`)
+
+- ``scrapy.utils.misc.md5sum()`` is now deprecated.
+ (:gh:`6264`)
+
+New features
+~~~~~~~~~~~~
+
+- ``scrapy.Spider.start_requests()`` can now yield items.
+ (:gh:`5289`, :gh:`6417`)
+
+ .. note:: Some spider middlewares may need to be updated for Scrapy 2.12
+ support before you can use them in combination with the ability to
+ yield items from ``start_requests()``.
+
+- Added a new :class:`~scrapy.http.Response` subclass,
+ :class:`~scrapy.http.JsonResponse`, for responses with a `JSON MIME type
+ <https://mimesniff.spec.whatwg.org/#json-mime-type>`_.
+ (:gh:`6069`, :gh:`6171`, :gh:`6174`)
+
+- The :class:`~scrapy.extensions.logstats.LogStats` extension now adds
+ ``items_per_minute`` and ``responses_per_minute`` to the :ref:`stats
+ <topics-stats>` when the spider closes.
+ (:gh:`4110`, :gh:`4111`)
+
+- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM` which allows closing the
+ spider if no items were scraped in a set amount of time.
+ (:gh:`6434`)
+
+- User-defined cookies can now include the ``secure`` field.
+ (:gh:`6357`)
+
+- Added component getters to :class:`~scrapy.crawler.Crawler`:
+ :meth:`~scrapy.crawler.Crawler.get_addon`,
+ :meth:`~scrapy.crawler.Crawler.get_downloader_middleware`,
+ :meth:`~scrapy.crawler.Crawler.get_extension`,
+ :meth:`~scrapy.crawler.Crawler.get_item_pipeline`,
+ :meth:`~scrapy.crawler.Crawler.get_spider_middleware`.
+ (:gh:`6181`)
+
+- Slot delay updates by the :ref:`AutoThrottle extension
+ <topics-autothrottle>` based on response latencies can now be disabled for
+ specific requests via the :reqmeta:`autothrottle_dont_adjust_delay` meta
+ key.
+ (:gh:`6246`, :gh:`6527`)
+
+- If :setting:`SPIDER_LOADER_WARN_ONLY` is set to ``True``,
+ :class:`~scrapy.spiderloader.SpiderLoader` does not raise
+ :exc:`SyntaxError` but emits a warning instead.
+ (:gh:`6483`, :gh:`6484`)
+
+- Added support for multiple-compressed responses (ones with several
+ encodings in the ``Content-Encoding`` header).
+ (:gh:`5143`, :gh:`5964`, :gh:`6063`)
+
+- Added support for multiple standard values in :setting:`REFERRER_POLICY`.
+ (:gh:`6381`)
+
+- Added support for brotlicffi_ (previously named brotlipy_). brotli_ is
+ still recommended but only brotlicffi_ works on PyPy.
+ (:gh:`6263`, :gh:`6269`)
+
+ .. _brotlicffi: https://github.com/python-hyper/brotlicffi
+
+- Added :class:`~scrapy.contracts.default.MetadataContract` that sets the
+ request meta.
+ (:gh:`6468`, :gh:`6469`)
+
+Improvements
+~~~~~~~~~~~~
+
+- Extended the list of file extensions that
+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ ignores by default.
+ (:gh:`6074`, :gh:`6125`)
+
+- :func:`scrapy.utils.httpobj.urlparse_cached` is now used in more places
+ instead of :func:`urllib.parse.urlparse`.
+ (:gh:`6228`, :gh:`6229`)
+
+Bug fixes
+~~~~~~~~~
+
+- :class:`~scrapy.pipelines.media.MediaPipeline` is now an abstract class and
+ its methods that were expected to be overridden in subclasses are now
+ abstract methods.
+ (:gh:`6365`, :gh:`6368`)
+
+- Fixed handling of invalid ``@``-prefixed lines in contract extraction.
+ (:gh:`6383`, :gh:`6388`)
+
+- Importing ``scrapy.extensions.telnet`` no longer installs the default
+ reactor.
+ (:gh:`6432`)
+
+- Reduced log verbosity for dropped requests that was increased in 2.11.2.
+ (:gh:`6433`, :gh:`6475`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Added ``SECURITY.md`` that documents the security policy.
+ (:gh:`5364`, :gh:`6051`)
+
+- Example code for :ref:`running Scrapy from a script <run-from-script>` no
+ longer imports ``twisted.internet.reactor`` at the top level, which caused
+ problems with non-default reactors when this code was used unmodified.
+ (:gh:`6361`, :gh:`6374`)
+
+- Documented the :class:`~scrapy.extensions.spiderstate.SpiderState`
+ extension.
+ (:gh:`6278`, :gh:`6522`)
+
+- Other documentation improvements and fixes.
+ (:gh:`5920`,
+ :gh:`6094`,
+ :gh:`6177`,
+ :gh:`6200`,
+ :gh:`6207`,
+ :gh:`6216`,
+ :gh:`6223`,
+ :gh:`6317`,
+ :gh:`6328`,
+ :gh:`6389`,
+ :gh:`6394`,
+ :gh:`6402`,
+ :gh:`6411`,
+ :gh:`6427`,
+ :gh:`6429`,
+ :gh:`6440`,
+ :gh:`6448`,
+ :gh:`6449`,
+ :gh:`6462`,
+ :gh:`6497`,
+ :gh:`6506`,
+ :gh:`6507`,
+ :gh:`6524`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Added ``py.typed``, in line with `PEP 561
+ <https://peps.python.org/pep-0561/>`_.
+ (:gh:`6058`, :gh:`6059`)
+
+- Fully covered the code with type hints (except for the most complicated
+ parts, mostly related to ``twisted.web.http`` and other Twisted parts
+ without type hints).
+ (:gh:`5989`,
+ :gh:`6097`,
+ :gh:`6127`,
+ :gh:`6129`,
+ :gh:`6130`,
+ :gh:`6133`,
+ :gh:`6143`,
+ :gh:`6191`,
+ :gh:`6268`,
+ :gh:`6274`,
+ :gh:`6275`,
+ :gh:`6276`,
+ :gh:`6279`,
+ :gh:`6325`,
+ :gh:`6326`,
+ :gh:`6333`,
+ :gh:`6335`,
+ :gh:`6336`,
+ :gh:`6337`,
+ :gh:`6341`,
+ :gh:`6353`,
+ :gh:`6356`,
+ :gh:`6370`,
+ :gh:`6371`,
+ :gh:`6384`,
+ :gh:`6385`,
+ :gh:`6387`,
+ :gh:`6391`,
+ :gh:`6395`,
+ :gh:`6414`,
+ :gh:`6422`,
+ :gh:`6460`,
+ :gh:`6466`,
+ :gh:`6472`,
+ :gh:`6494`,
+ :gh:`6498`,
+ :gh:`6516`)
+
+- Improved Bandit_ checks.
+ (:gh:`6260`, :gh:`6264`, :gh:`6265`)
+
+- Added pyupgrade_ to the ``pre-commit`` configuration.
+ (:gh:`6392`)
+
+ .. _pyupgrade: https://github.com/asottile/pyupgrade
+
+- Added ``flake8-bugbear``, ``flake8-comprehensions``, ``flake8-debugger``,
+ ``flake8-docstrings``, ``flake8-string-format`` and
+ ``flake8-type-checking`` to the ``pre-commit`` configuration.
+ (:gh:`6406`, :gh:`6413`)
+
+- CI and test improvements and fixes.
+ (:gh:`5285`,
+ :gh:`5454`,
+ :gh:`5997`,
+ :gh:`6078`,
+ :gh:`6084`,
+ :gh:`6087`,
+ :gh:`6132`,
+ :gh:`6153`,
+ :gh:`6154`,
+ :gh:`6201`,
+ :gh:`6231`,
+ :gh:`6232`,
+ :gh:`6235`,
+ :gh:`6236`,
+ :gh:`6242`,
+ :gh:`6245`,
+ :gh:`6253`,
+ :gh:`6258`,
+ :gh:`6259`,
+ :gh:`6270`,
+ :gh:`6272`,
+ :gh:`6286`,
+ :gh:`6290`,
+ :gh:`6296`
+ :gh:`6367`,
+ :gh:`6372`,
+ :gh:`6403`,
+ :gh:`6416`,
+ :gh:`6435`,
+ :gh:`6489`,
+ :gh:`6501`,
+ :gh:`6504`,
+ :gh:`6511`,
+ :gh:`6543`,
+ :gh:`6545`)
+
+- Code cleanups.
+ (:gh:`6196`,
+ :gh:`6197`,
+ :gh:`6198`,
+ :gh:`6199`,
+ :gh:`6254`,
+ :gh:`6257`,
+ :gh:`6285`,
+ :gh:`6305`,
+ :gh:`6343`,
+ :gh:`6349`,
+ :gh:`6386`,
+ :gh:`6415`,
+ :gh:`6463`,
+ :gh:`6470`,
+ :gh:`6499`,
+ :gh:`6505`,
+ :gh:`6510`,
+ :gh:`6531`,
+ :gh:`6542`)
+
+Other
+~~~~~
+
+- Issue tracker improvements. (:gh:`6066`)
+
+
+.. _release-2.11.2:
+
+Scrapy 2.11.2 (2024-05-14)
+--------------------------
+
+Security bug fixes
+~~~~~~~~~~~~~~~~~~
+
+- Redirects to non-HTTP protocols are no longer followed. Please, see the
+ `23j4-mw76-5v7h security advisory`_ for more information. (:gh:`457`)
+
+ .. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h
+
+- The ``Authorization`` header is now dropped on redirects to a different
+ scheme (``http://`` or ``https://``) or port, even if the domain is the
+ same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more
+ information.
+
+ .. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f
+
+- When using system proxy settings that are different for ``http://`` and
+ ``https://``, redirects to a different URL scheme will now also trigger the
+ corresponding change in proxy settings for the redirected request. Please,
+ see the `jm3v-qxmh-hxwv security advisory`_ for more information.
+ (:gh:`767`)
+
+ .. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv
+
+- :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>` is now
+ enforced for all requests, and not only requests from spider callbacks.
+ (:gh:`1042`, :gh:`2241`, :gh:`6358`)
+
+- :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML
+ entities. (:gh:`6265`)
+
+- defusedxml_ is now used to make
+ :class:`scrapy.http.request.rpc.XmlRpcRequest` more secure.
+ (:gh:`6250`, :gh:`6251`)
+
+ .. _defusedxml: https://github.com/tiran/defusedxml
+
+Deprecations
+~~~~~~~~~~~~
+
+- ``scrapy.spidermiddlewares.offsite.OffsiteMiddleware`` (a spider
+ middleware) is now deprecated and not enabled by default. The new
+ downloader middleware with the same functionality,
+ :class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, is enabled
+ instead.
+ (:gh:`2241`, :gh:`6358`)
+
+
+Bug fixes
+~~~~~~~~~
+
+- Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
+ favor of brotli_. (:gh:`6261`)
+
+ .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
+ instead if you can.
+
+- Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This
+ prevents
+ :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from
+ following redirects that would not be followed by web browsers with
+ JavaScript enabled. (:gh:`6342`, :gh:`6347`)
+
+- During :ref:`feed export <topics-feed-exports>`, do not close the
+ underlying file from :ref:`built-in post-processing plugins
+ <builtin-plugins>`.
+ (:gh:`5932`, :gh:`6178`, :gh:`6239`)
+
+- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ now properly applies the ``unique`` and ``canonicalize`` parameters.
+ (:gh:`3273`, :gh:`6221`)
+
+- Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty
+ string. (:gh:`6121`, :gh:`6124`)
+
+- Fix :attr:`Spider.logger <scrapy.Spider.logger>` not logging custom extra
+ information. (:gh:`6323`, :gh:`6324`)
+
+- ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing
+ the UTF-8-compatible (e.g. ASCII) parts of the document.
+ (:gh:`6292`, :gh:`6298`)
+
+- :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an
+ exception if ``default`` is ``None``.
+ (:gh:`6308`, :gh:`6310`)
+
+- :class:`~scrapy.Selector` now uses
+ :func:`scrapy.utils.response.get_base_url` to determine the base URL of a
+ given :class:`~scrapy.http.Response`. (:gh:`6265`)
+
+- The :meth:`media_to_download` method of :ref:`media pipelines
+ <topics-media-pipeline>` now logs exceptions before stripping them.
+ (:gh:`5067`, :gh:`5068`)
+
+- When passing a callback to the :command:`parse` command, build the callback
+ callable with the right signature.
+ (:gh:`6182`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`.
+ (:gh:`6203`, :gh:`6208`)
+
+- Document that :attr:`scrapy.Selector.type` can be ``"json"``.
+ (:gh:`6328`, :gh:`6334`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Make builds reproducible. (:gh:`5019`, :gh:`6322`)
+
+- Packaging and test fixes.
+ (:gh:`6286`, :gh:`6290`, :gh:`6312`, :gh:`6316`, :gh:`6344`)
+
+
+.. _release-2.11.1:
+
+Scrapy 2.11.1 (2024-02-14)
+--------------------------
+
+Highlights:
+
+- Security bug fixes.
+
+- Support for Twisted >= 23.8.0.
+
+- Documentation improvements.
+
+Security bug fixes
+~~~~~~~~~~~~~~~~~~
+
+- Addressed `ReDoS vulnerabilities`_:
+
+ - ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of
+ :func:`~scrapy.utils.iterators.xmliter_lxml`, which
+ :class:`~scrapy.spiders.XMLFeedSpider` now uses.
+
+ To minimize the impact of this change on existing code,
+ :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
+ the node namespace with a prefix in the node name, and big files with
+ highly nested trees when using libxml2 2.7+.
+
+ - Fixed regular expressions in the implementation of the
+ :func:`~scrapy.utils.response.open_in_browser` function.
+
+ Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.
+
+ .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
+ .. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9
+
+- :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
+ to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
+ advisory`_ for more information.
+
+ .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7
+
+- Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
+ deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
+ removed.
+
+- The ``Authorization`` header is now dropped on redirects to a different
+ domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
+ information.
+
+ .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- The Twisted dependency is no longer restricted to < 23.8.0. (:gh:`6024`,
+ :gh:`6064`, :gh:`6142`)
+
+Bug fixes
+~~~~~~~~~
+
+- The OS signal handling code was refactored to no longer use private Twisted
+ functions. (:gh:`6024`, :gh:`6064`, :gh:`6112`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
+ changes made in the 2.11.0 release. (:gh:`6057`, :gh:`6147`)
+
+- Extended documentation for :attr:`.Request.meta`.
+ (:gh:`5565`)
+
+- Fixed the :reqmeta:`dont_merge_cookies` documentation. (:gh:`5936`,
+ :gh:`6077`)
+
+- Added a link to Zyte's export guides to the :ref:`feed exports
+ <topics-feed-exports>` documentation. (:gh:`6183`)
+
+- Added a missing note about backward-incompatible changes in
+ :class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes.
+ (:gh:`6060`, :gh:`6081`)
+
+- Added a missing note about removing the deprecated
+ ``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes.
+ (:gh:`6056`, :gh:`6061`)
+
+- Other documentation improvements. (:gh:`6128`, :gh:`6144`,
+ :gh:`6163`, :gh:`6190`, :gh:`6192`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Added Python 3.12 to the CI configuration, re-enabled tests that were
+ disabled when the pre-release support was added. (:gh:`5985`,
+ :gh:`6083`, :gh:`6098`)
+
+- Fixed a test issue on PyPy 7.3.14. (:gh:`6204`, :gh:`6205`)
+
+
+.. _release-2.11.0:
+
+Scrapy 2.11.0 (2023-09-18)
+--------------------------
+
+Highlights:
+
+- Spiders can now modify :ref:`settings <topics-settings>` in their
+ :meth:`~scrapy.Spider.from_crawler` methods, e.g. based on :ref:`spider
+ arguments <spiderargs>`.
+
+- Periodic logging of stats.
+
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Most of the initialization of :class:`scrapy.crawler.Crawler` instances is
+ now done in :meth:`~scrapy.crawler.Crawler.crawl`, so the state of
+ instances before that method is called is now different compared to older
+ Scrapy versions. We do not recommend using the
+ :class:`~scrapy.crawler.Crawler` instances before
+ :meth:`~scrapy.crawler.Crawler.crawl` is called. (:gh:`6038`)
+
+- :meth:`scrapy.Spider.from_crawler` is now called before the initialization
+ of various components previously initialized in
+ :meth:`scrapy.crawler.Crawler.__init__` and before the settings are
+ finalized and frozen. This change was needed to allow changing the settings
+ in :meth:`scrapy.Spider.from_crawler`. If you want to access the final
+ setting values and the initialized :class:`~scrapy.crawler.Crawler`
+ attributes in the spider code as early as possible you can do this in
+ ``scrapy.Spider.start_requests()`` or in a handler of the
+ :signal:`engine_started` signal. (:gh:`6038`)
+
+- The :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method now
+ requires the response to be in a valid JSON encoding (UTF-8, UTF-16, or
+ UTF-32). If you need to deal with JSON documents in an invalid encoding,
+ use ``json.loads(response.text)`` instead. (:gh:`6016`)
+
+- :class:`~scrapy.exporters.PythonItemExporter` used the binary output by
+ default but it no longer does. (:gh:`6006`, :gh:`6007`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- Removed the binary export mode of
+ :class:`~scrapy.exporters.PythonItemExporter`, deprecated in Scrapy 1.1.0.
+ (:gh:`6006`, :gh:`6007`)
+
+ .. note:: If you are using this Scrapy version on Scrapy Cloud with a stack
+ that includes an older Scrapy version and get a "TypeError:
+ Unexpected options: binary" error, you may need to add
+ ``scrapinghub-entrypoint-scrapy >= 0.14.1`` to your project
+ requirements or switch to a stack that includes Scrapy 2.11.
+
+- Removed the ``CrawlerRunner.spiders`` attribute, deprecated in Scrapy
+ 1.0.0, use :attr:`CrawlerRunner.spider_loader
+ <scrapy.crawler.CrawlerRunner.spider_loader>` instead. (:gh:`6010`)
+
+- The :func:`scrapy.utils.response.response_httprepr` function, deprecated in
+ Scrapy 2.6.0, has now been removed. (:gh:`6111`)
+
+Deprecations
+~~~~~~~~~~~~
+
+- Running :meth:`~scrapy.crawler.Crawler.crawl` more than once on the same
+ :class:`scrapy.crawler.Crawler` instance is now deprecated. (:gh:`1587`,
+ :gh:`6040`)
+
+New features
+~~~~~~~~~~~~
+
+- Spiders can now modify settings in their
+ :meth:`~scrapy.Spider.from_crawler` method, e.g. based on :ref:`spider
+ arguments <spiderargs>`. (:gh:`1305`, :gh:`1580`, :gh:`2392`,
+ :gh:`3663`, :gh:`6038`)
+
+- Added the :class:`~scrapy.extensions.periodic_log.PeriodicLog` extension
+ which can be enabled to log stats and/or their differences periodically.
+ (:gh:`5926`)
+
+- Optimized the memory usage in :meth:`TextResponse.json
+ <scrapy.http.TextResponse.json>` by removing unnecessary body decoding.
+ (:gh:`5968`, :gh:`6016`)
+
+- Links to ``.webp`` files are now ignored by :ref:`link extractors
+ <topics-link-extractors>`. (:gh:`6021`)
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed logging enabled add-ons. (:gh:`6036`)
+
+- Fixed :class:`~scrapy.mail.MailSender` producing invalid message bodies
+ when the ``charset`` argument is passed to
+ :meth:`~scrapy.mail.MailSender.send`. (:gh:`5096`, :gh:`5118`)
+
+- Fixed an exception when accessing ``self.EXCEPTIONS_TO_RETRY`` from a
+ subclass of :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
+ (:gh:`6049`, :gh:`6050`)
+
+- :meth:`scrapy.settings.BaseSettings.getdictorlist`, used to parse
+ :setting:`FEED_EXPORT_FIELDS`, now handles tuple values. (:gh:`6011`,
+ :gh:`6013`)
+
+- Calls to ``datetime.utcnow()``, no longer recommended to be used, have been
+ replaced with calls to ``datetime.now()`` with a timezone. (:gh:`6014`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Updated a deprecated function call in a pipeline example. (:gh:`6008`,
+ :gh:`6009`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Extended typing hints. (:gh:`6003`, :gh:`6005`, :gh:`6031`,
+ :gh:`6034`)
+
+- Pinned brotli_ to 1.0.9 for the PyPy tests as 1.1.0 breaks them.
+ (:gh:`6044`, :gh:`6045`)
+
+- Other CI and pre-commit improvements. (:gh:`6002`, :gh:`6013`,
+ :gh:`6046`)
+
+.. _release-2.10.1:
+
+Scrapy 2.10.1 (2023-08-30)
+--------------------------
+
+Marked ``Twisted >= 23.8.0`` as unsupported. (:gh:`6024`, :gh:`6026`)
+
+.. _release-2.10.0:
+
+Scrapy 2.10.0 (2023-08-04)
+--------------------------
+
+Highlights:
+
+- Added Python 3.12 support, dropped Python 3.7 support.
+
+- The new add-ons framework simplifies configuring 3rd-party components that
+ support it.
+
+- Exceptions to retry can now be configured.
+
+- Many fixes and improvements for feed exports.
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Dropped support for Python 3.7. (:gh:`5953`)
+
+- Added support for the upcoming Python 3.12. (:gh:`5984`)
+
+- Minimum versions increased for these dependencies:
+
+ - lxml_: 4.3.0 → 4.4.1
+
+ - cryptography_: 3.4.6 → 36.0.0
+
+- ``pkg_resources`` is no longer used. (:gh:`5956`, :gh:`5958`)
+
+- boto3_ is now recommended instead of botocore_ for exporting to S3.
+ (:gh:`5833`).
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- The value of the :setting:`FEED_STORE_EMPTY` setting is now ``True``
+ instead of ``False``. In earlier Scrapy versions empty files were created
+ even when this setting was ``False`` (which was a bug that is now fixed),
+ so the new default should keep the old behavior. (:gh:`872`,
+ :gh:`5847`)
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
+ returning ``None`` or modifying the ``params`` input parameter, deprecated
+ in Scrapy 2.6, is no longer supported. (:gh:`5994`, :gh:`5996`)
+
+- The ``scrapy.utils.reqser`` module, deprecated in Scrapy 2.6, is removed.
+ (:gh:`5994`, :gh:`5996`)
+
+- The ``scrapy.squeues`` classes ``PickleFifoDiskQueueNonRequest``,
+ ``PickleLifoDiskQueueNonRequest``, ``MarshalFifoDiskQueueNonRequest``,
+ and ``MarshalLifoDiskQueueNonRequest``, deprecated in
+ Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5996`)
+
+- The property ``open_spiders`` and the methods ``has_capacity`` and
+ ``schedule`` of :class:`scrapy.core.engine.ExecutionEngine`,
+ deprecated in Scrapy 2.6, are removed. (:gh:`5994`, :gh:`5998`)
+
+- Passing a ``spider`` argument to the
+ :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`,
+ :meth:`~scrapy.core.engine.ExecutionEngine.crawl` and
+ :meth:`~scrapy.core.engine.ExecutionEngine.download` methods of
+ :class:`scrapy.core.engine.ExecutionEngine`, deprecated in Scrapy 2.6, is
+ no longer supported. (:gh:`5994`, :gh:`5998`)
+
+Deprecations
+~~~~~~~~~~~~
+
+- :class:`scrapy.utils.datatypes.CaselessDict` is deprecated, use
+ :class:`scrapy.utils.datatypes.CaseInsensitiveDict` instead.
+ (:gh:`5146`)
+
+- Passing the ``custom`` argument to
+ :func:`scrapy.utils.conf.build_component_list` is deprecated, it was used
+ in the past to merge ``FOO`` and ``FOO_BASE`` setting values but now Scrapy
+ uses :func:`scrapy.settings.BaseSettings.getwithbase` to do the same.
+ Code that uses this argument and cannot be switched to ``getwithbase()``
+ can be switched to merging the values explicitly. (:gh:`5726`,
+ :gh:`5923`)
+
+New features
+~~~~~~~~~~~~
+
+- Added support for :ref:`Scrapy add-ons <topics-addons>`. (:gh:`5950`)
+
+- Added the :setting:`RETRY_EXCEPTIONS` setting that configures which
+ exceptions will be retried by
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware`.
+ (:gh:`2701`, :gh:`5929`)
+
+- Added the possiiblity to close the spider if no items were produced in the
+ specified time, configured by :setting:`CLOSESPIDER_TIMEOUT_NO_ITEM`.
+ (:gh:`5979`)
+
+- Added support for the :setting:`AWS_REGION_NAME` setting to feed exports.
+ (:gh:`5980`)
+
+- Added support for using :class:`pathlib.Path` objects that refer to
+ absolute Windows paths in the :setting:`FEEDS` setting. (:gh:`5939`)
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed creating empty feeds even with ``FEED_STORE_EMPTY=False``.
+ (:gh:`872`, :gh:`5847`)
+
+- Fixed using absolute Windows paths when specifying output files.
+ (:gh:`5969`, :gh:`5971`)
+
+- Fixed problems with uploading large files to S3 by switching to multipart
+ uploads (requires boto3_). (:gh:`960`, :gh:`5735`, :gh:`5833`)
+
+- Fixed the JSON exporter writing extra commas when some exceptions occur.
+ (:gh:`3090`, :gh:`5952`)
+
+- Fixed the "read of closed file" error in the CSV exporter. (:gh:`5043`,
+ :gh:`5705`)
+
+- Fixed an error when a component added by the class object throws
+ :exc:`~scrapy.exceptions.NotConfigured` with a message. (:gh:`5950`,
+ :gh:`5992`)
+
+- Added the missing :meth:`scrapy.settings.BaseSettings.pop` method.
+ (:gh:`5959`, :gh:`5960`, :gh:`5963`)
+
+- Added :class:`~scrapy.utils.datatypes.CaseInsensitiveDict` as a replacement
+ for :class:`~scrapy.utils.datatypes.CaselessDict` that fixes some API
+ inconsistencies. (:gh:`5146`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Documented :meth:`scrapy.Spider.update_settings`. (:gh:`5745`,
+ :gh:`5846`)
+
+- Documented possible problems with early Twisted reactor installation and
+ their solutions. (:gh:`5981`, :gh:`6000`)
+
+- Added examples of making additional requests in callbacks. (:gh:`5927`)
+
+- Improved the feed export docs. (:gh:`5579`, :gh:`5931`)
+
+- Clarified the docs about request objects on redirection. (:gh:`5707`,
+ :gh:`5937`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Added support for running tests against the installed Scrapy version.
+ (:gh:`4914`, :gh:`5949`)
+
+- Extended typing hints. (:gh:`5925`, :gh:`5977`)
+
+- Fixed the ``test_utils_asyncio.AsyncioTest.test_set_asyncio_event_loop``
+ test. (:gh:`5951`)
+
+- Fixed the ``test_feedexport.BatchDeliveriesTest.test_batch_path_differ``
+ test on Windows. (:gh:`5847`)
+
+- Enabled CI runs for Python 3.11 on Windows. (:gh:`5999`)
+
+- Simplified skipping tests that depend on ``uvloop``. (:gh:`5984`)
+
+- Fixed the ``extra-deps-pinned`` tox env. (:gh:`5948`)
+
+- Implemented cleanups. (:gh:`5965`, :gh:`5986`)
+
+.. _release-2.9.0:
+
+Scrapy 2.9.0 (2023-05-08)
+-------------------------
+
+Highlights:
+
+- Per-domain download settings.
+- Compatibility with new cryptography_ and new parsel_.
+- JMESPath selectors from the new parsel_.
+- Bug fixes.
+
+Deprecations
+~~~~~~~~~~~~
+
+- :class:`scrapy.extensions.feedexport._FeedSlot` is renamed to
+ :class:`scrapy.extensions.feedexport.FeedSlot` and the old name is
+ deprecated. (:gh:`5876`)
+
+New features
+~~~~~~~~~~~~
+
+- Settings corresponding to :setting:`DOWNLOAD_DELAY`,
+ :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` and
+ ``RANDOMIZE_DOWNLOAD_DELAY`` can now be set on a per-domain basis
+ via the new :setting:`DOWNLOAD_SLOTS` setting. (:gh:`5328`)
+
+- Added :meth:`.TextResponse.jmespath`, a shortcut for JMESPath selectors
+ available since parsel_ 1.8.1. (:gh:`5894`, :gh:`5915`)
+
+- Added :signal:`feed_slot_closed` and :signal:`feed_exporter_closed`
+ signals. (:gh:`5876`)
+
+- Added :func:`scrapy.utils.request.request_to_curl`, a function to produce a
+ curl command from a :class:`~scrapy.Request` object. (:gh:`5892`)
+
+- Values of :setting:`FILES_STORE` and :setting:`IMAGES_STORE` can now be
+ :class:`pathlib.Path` instances. (:gh:`5801`)
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed a warning with Parsel 1.8.1+. (:gh:`5903`, :gh:`5918`)
+
+- Fixed an error when using feed postprocessing with S3 storage.
+ (:gh:`5500`, :gh:`5581`)
+
+- Added the missing :meth:`scrapy.settings.BaseSettings.setdefault` method.
+ (:gh:`5811`, :gh:`5821`)
+
+- Fixed an error when using cryptography_ 40.0.0+ and
+ :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` is enabled.
+ (:gh:`5857`, :gh:`5858`)
+
+- The checksums returned by :class:`~scrapy.pipelines.files.FilesPipeline`
+ for files on Google Cloud Storage are no longer Base64-encoded.
+ (:gh:`5874`, :gh:`5891`)
+
+- :func:`scrapy.utils.request.request_from_curl` now supports $-prefixed
+ string values for the curl ``--data-raw`` argument, which are produced by
+ browsers for data that includes certain symbols. (:gh:`5899`,
+ :gh:`5901`)
+
+- The :command:`parse` command now also works with async generator callbacks.
+ (:gh:`5819`, :gh:`5824`)
+
+- The :command:`genspider` command now properly works with HTTPS URLs.
+ (:gh:`3553`, :gh:`5808`)
+
+- Improved handling of asyncio loops. (:gh:`5831`, :gh:`5832`)
+
+- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ now skips certain malformed URLs instead of raising an exception.
+ (:gh:`5881`)
+
+- :func:`scrapy.utils.python.get_func_args` now supports more types of
+ callables. (:gh:`5872`, :gh:`5885`)
+
+- Fixed an error when processing non-UTF8 values of ``Content-Type`` headers.
+ (:gh:`5914`, :gh:`5917`)
+
+- Fixed an error breaking user handling of send failures in
+ :meth:`scrapy.mail.MailSender.send`. (:gh:`1611`, :gh:`5880`)
+
+Documentation
+~~~~~~~~~~~~~
+
+- Expanded contributing docs. (:gh:`5109`, :gh:`5851`)
+
+- Added blacken-docs_ to pre-commit and reformatted the docs with it.
+ (:gh:`5813`, :gh:`5816`)
+
+- Fixed a JS issue. (:gh:`5875`, :gh:`5877`)
+
+- Fixed ``make htmlview``. (:gh:`5878`, :gh:`5879`)
+
+- Fixed typos and other small errors. (:gh:`5827`, :gh:`5839`,
+ :gh:`5883`, :gh:`5890`, :gh:`5895`, :gh:`5904`)
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Extended typing hints. (:gh:`5805`, :gh:`5889`, :gh:`5896`)
+
+- Tests for most of the examples in the docs are now run as a part of CI,
+ found problems were fixed. (:gh:`5816`, :gh:`5826`, :gh:`5919`)
+
+- Removed usage of deprecated Python classes. (:gh:`5849`)
+
+- Silenced ``include-ignored`` warnings from coverage. (:gh:`5820`)
+
+- Fixed a random failure of the ``test_feedexport.test_batch_path_differ``
+ test. (:gh:`5855`, :gh:`5898`)
+
+- Updated docstrings to match output produced by parsel_ 1.8.1 so that they
+ don't cause test failures. (:gh:`5902`, :gh:`5919`)
+
+- Other CI and pre-commit improvements. (:gh:`5802`, :gh:`5823`,
+ :gh:`5908`)
+
+.. _blacken-docs: https://github.com/adamchainz/blacken-docs
+
+.. _release-2.8.0:
+
+Scrapy 2.8.0 (2023-02-02)
+-------------------------
+
+This is a maintenance release, with minor features, bug fixes, and cleanups.
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
+ been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
+ :class:`~gzip.GzipFile` instead.
+ (:gh:`5719`)
+
+- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
+ 2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
+ instead.
+ (:gh:`5719`)
+
+- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
+ 2.0, has now been removed. Use
+ :meth:`~scrapy.utils.python.MutableChain.__next__` instead.
+ (:gh:`5719`)
+
+- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
+ in Scrapy 2.0, has now been removed. Use
+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ instead.
+ (:gh:`5720`)
+
+- Support for using environment variables prefixed with ``SCRAPY_`` to
+ override settings, deprecated in Scrapy 2.0, has now been removed.
+ (:gh:`5724`)
+
+- Support for the ``noconnect`` query string argument in proxy URLs,
+ deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
+ to need it to work fine without it.
+ (:gh:`5731`)
+
+- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
+ 2.3, has now been removed.
+ (:gh:`5719`)
+
+- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
+ has now been removed.
+ (:gh:`5719`)
+
+- The ``scrapy.utils.boto.is_botocore()`` function, deprecated in Scrapy 2.4,
+ has now been removed.
+ (:gh:`5719`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
+ (:gh:`5368`, :gh:`5489`)
+
+- :meth:`ImagesPipeline.convert_image
+ <scrapy.pipelines.images.ImagesPipeline.convert_image>` must now accept a
+ ``response_body`` parameter.
+ (:gh:`3055`, :gh:`3689`, :gh:`4753`)
+
+
+New features
+~~~~~~~~~~~~
+
+- Applied black_ coding style to files generated with the
+ :command:`genspider` and :command:`startproject` commands.
+ (:gh:`5809`, :gh:`5814`)
+
+ .. _black: https://black.readthedocs.io/en/stable/
+
+- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
+ ``settings.py`` file that the :command:`startproject` command generates.
+ With this value, JSON exports won’t force the use of escape sequences for
+ non-ASCII characters.
+ (:gh:`5797`, :gh:`5800`)
+
+- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
+ peak memory usage during checks, and the binary unit MiB is now used to
+ avoid confusion.
+ (:gh:`5717`, :gh:`5722`, :gh:`5727`)
+
+- The ``callback`` parameter of :class:`~scrapy.Request` can now be set
+ to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
+ ``None``, as the latter indicates that the default spider callback
+ (:meth:`~scrapy.Spider.parse`) is to be used.
+ (:gh:`5798`)
+
+
+Bug fixes
+~~~~~~~~~
+
+- Enabled unsafe legacy SSL renegotiation to fix access to some outdated
+ websites.
+ (:gh:`5491`, :gh:`5790`)
+
+- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
+ better.
+ (:gh:`5386`, :gh:`5406`)
+
+- Fixed the :meth:`finish_exporting` method of :ref:`item exporters
+ <topics-exporters>` not being called for empty files.
+ (:gh:`5537`, :gh:`5758`)
+
+- Fixed HTTP/2 responses getting only the last value for a header when
+ multiple headers with the same name are received.
+ (:gh:`5777`)
+
+- Fixed an exception raised by the :command:`shell` command on some cases
+ when :ref:`using asyncio <using-asyncio>`.
+ (:gh:`5740`, :gh:`5742`, :gh:`5748`, :gh:`5759`, :gh:`5760`,
+ :gh:`5771`)
+
+- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
+ (``cb_kwargs``) added to a request in the ``process_request`` callback of a
+ :class:`~scrapy.spiders.Rule` will no longer be ignored.
+ (:gh:`5699`)
+
+- The :ref:`images pipeline <images-pipeline>` no longer re-encodes JPEG
+ files.
+ (:gh:`3055`, :gh:`3689`, :gh:`4753`)
+
+- Fixed the handling of transparent WebP images by the :ref:`images pipeline
+ <images-pipeline>`.
+ (:gh:`3072`, :gh:`5766`, :gh:`5767`)
+
+- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
+ (Ctrl+C).
+ (:gh:`2918`)
+
+- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ with ``unique=False`` no longer filters out links that have identical URL
+ *and* text.
+ (:gh:`3798`, :gh:`3799`, :gh:`4695`, :gh:`5458`)
+
+- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
+ ignores URL protocols that do not support ``robots.txt`` (``data://``,
+ ``file://``).
+ (:gh:`5807`)
+
+- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
+ (:gh:`5753`, :gh:`5754`)
+
+- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
+ line.
+ (:gh:`5709`, :gh:`5711`, :gh:`5712`)
+
+- Made the active project indication in the output of :ref:`commands
+ <topics-commands>` more clear.
+ (:gh:`5715`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+- Documented how to :ref:`debug spiders from Visual Studio Code
+ <debug-vscode>`.
+ (:gh:`5721`)
+
+- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
+ (:gh:`5083`, :gh:`5540`)
+
+- Improved consistency.
+ (:gh:`5761`)
+
+- Fixed typos.
+ (:gh:`5714`, :gh:`5744`, :gh:`5764`)
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Applied :ref:`black coding style <coding-style>`, sorted import statements,
+ and introduced :ref:`pre-commit <scrapy-pre-commit>`.
+ (:gh:`4654`, :gh:`4658`, :gh:`5734`, :gh:`5737`, :gh:`5806`,
+ :gh:`5810`)
+
+- Switched from :mod:`os.path` to :mod:`pathlib`.
+ (:gh:`4916`, :gh:`4497`, :gh:`5682`)
+
+- Addressed many issues reported by Pylint.
+ (:gh:`5677`)
+
+- Improved code readability.
+ (:gh:`5736`)
+
+- Improved package metadata.
+ (:gh:`5768`)
+
+- Removed direct invocations of ``setup.py``.
+ (:gh:`5774`, :gh:`5776`)
+
+- Removed unnecessary :class:`~collections.OrderedDict` usages.
+ (:gh:`5795`)
+
+- Removed unnecessary ``__str__`` definitions.
+ (:gh:`5150`)
+
+- Removed obsolete code and comments.
+ (:gh:`5725`, :gh:`5729`, :gh:`5730`, :gh:`5732`)
+
+- Fixed test and CI issues.
+ (:gh:`5749`, :gh:`5750`, :gh:`5756`, :gh:`5762`, :gh:`5765`,
+ :gh:`5780`, :gh:`5781`, :gh:`5782`, :gh:`5783`, :gh:`5785`,
+ :gh:`5786`)
+
+
+.. _release-2.7.1:
+
+Scrapy 2.7.1 (2022-11-02)
+-------------------------
+
+New features
+~~~~~~~~~~~~
+
+- Relaxed the restriction introduced in 2.6.2 so that the
+ ``Proxy-Authorization`` header can again be set explicitly, as long as the
+ proxy URL in the :reqmeta:`proxy` metadata has no other credentials, and
+ for as long as that proxy URL remains the same; this restores compatibility
+ with scrapy-zyte-smartproxy 2.1.0 and older (:gh:`5626`).
+
+Bug fixes
+~~~~~~~~~
+
+- Using ``-O``/``--overwrite-output`` and ``-t``/``--output-format`` options
+ together now produces an error instead of ignoring the former option
+ (:gh:`5516`, :gh:`5605`).
+
+- Replaced deprecated :mod:`asyncio` APIs that implicitly use the current
+ event loop with code that explicitly requests a loop from the event loop
+ policy (:gh:`5685`, :gh:`5689`).
+
+- Fixed uses of deprecated Scrapy APIs in Scrapy itself (:gh:`5588`,
+ :gh:`5589`).
+
+- Fixed uses of a deprecated Pillow API (:gh:`5684`, :gh:`5692`).
+
+- Improved code that checks if generators return values, so that it no longer
+ fails on decorated methods and partial methods (:gh:`5323`,
+ :gh:`5592`, :gh:`5599`, :gh:`5691`).
+
+Documentation
+~~~~~~~~~~~~~
+
+- Upgraded the Code of Conduct to Contributor Covenant v2.1 (:gh:`5698`).
+
+- Fixed typos (:gh:`5681`, :gh:`5694`).
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Re-enabled some erroneously disabled flake8 checks (:gh:`5688`).
+
+- Ignored harmless deprecation warnings from :mod:`typing` in tests
+ (:gh:`5686`, :gh:`5697`).
+
+- Modernized our CI configuration (:gh:`5695`, :gh:`5696`).
+
+
+.. _release-2.7.0:
+
+Scrapy 2.7.0 (2022-10-17)
+-------------------------
+
+Highlights:
+
+- Added Python 3.11 support, dropped Python 3.6 support
+- Improved support for :ref:`asynchronous callbacks <topics-coroutines>`
+- :ref:`Asyncio support <using-asyncio>` is enabled by default on new
+ projects
+- Output names of item fields can now be arbitrary strings
+- Centralized :ref:`request fingerprinting <request-fingerprints>`
+ configuration is now possible
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+Python 3.7 or greater is now required; support for Python 3.6 has been dropped.
+Support for the upcoming Python 3.11 has been added.
+
+The minimum required version of some dependencies has changed as well:
+
+- lxml_: 3.5.0 → 4.3.0
+
+- Pillow_ (:ref:`images pipeline <images-pipeline>`): 4.0.0 → 7.1.0
+
+- zope.interface_: 5.0.0 → 5.1.0
+
+(:gh:`5512`, :gh:`5514`, :gh:`5524`, :gh:`5563`, :gh:`5664`,
+:gh:`5670`, :gh:`5678`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+- :meth:`ImagesPipeline.thumb_path
+ <scrapy.pipelines.images.ImagesPipeline.thumb_path>` must now accept an
+ ``item`` parameter (:gh:`5504`, :gh:`5508`).
+
+- The ``scrapy.downloadermiddlewares.decompression`` module is now
+ deprecated (:gh:`5546`, :gh:`5547`).
+
+
+New features
+~~~~~~~~~~~~
+
+- The
+ :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
+ method of :ref:`spider middlewares <topics-spider-middleware>` can now be
+ defined as an :term:`asynchronous generator` (:gh:`4978`).
+
+- The output of :class:`~scrapy.Request` callbacks defined as
+ :ref:`coroutines <topics-coroutines>` is now processed asynchronously
+ (:gh:`4978`).
+
+- :class:`~scrapy.spiders.crawl.CrawlSpider` now supports :ref:`asynchronous
+ callbacks <topics-coroutines>` (:gh:`5657`).
+
+- New projects created with the :command:`startproject` command have
+ :ref:`asyncio support <using-asyncio>` enabled by default (:gh:`5590`,
+ :gh:`5679`).
+
+- The :setting:`FEED_EXPORT_FIELDS` setting can now be defined as a
+ dictionary to customize the output name of item fields, lifting the
+ restriction that required output names to be valid Python identifiers, e.g.
+ preventing them to have whitespace (:gh:`1008`, :gh:`3266`,
+ :gh:`3696`).
+
+- You can now customize :ref:`request fingerprinting <request-fingerprints>`
+ through the new :setting:`REQUEST_FINGERPRINTER_CLASS` setting, instead of
+ having to change it on every Scrapy component that relies on request
+ fingerprinting (:gh:`900`, :gh:`3420`, :gh:`4113`, :gh:`4762`,
+ :gh:`4524`).
+
+- ``jsonl`` is now supported and encouraged as a file extension for `JSON
+ Lines`_ files (:gh:`4848`).
+
+ .. _JSON Lines: https://jsonlines.org/
+
+- :meth:`ImagesPipeline.thumb_path
+ <scrapy.pipelines.images.ImagesPipeline.thumb_path>` now receives the
+ source :ref:`item <topics-items>` (:gh:`5504`, :gh:`5508`).
+
+
+Bug fixes
+~~~~~~~~~
+
+- When using Google Cloud Storage with a :ref:`media pipeline
+ <topics-media-pipeline>`, :setting:`FILES_EXPIRES` now also works when
+ :setting:`FILES_STORE` does not point at the root of your Google Cloud
+ Storage bucket (:gh:`5317`, :gh:`5318`).
+
+- The :command:`parse` command now supports :ref:`asynchronous callbacks
+ <topics-coroutines>` (:gh:`5424`, :gh:`5577`).
+
+- When using the :command:`parse` command with a URL for which there is no
+ available spider, an exception is no longer raised (:gh:`3264`,
+ :gh:`3265`, :gh:`5375`, :gh:`5376`, :gh:`5497`).
+
+- :class:`~scrapy.http.TextResponse` now gives higher priority to the `byte
+ order mark`_ when determining the text encoding of the response body,
+ following the `HTML living standard`_ (:gh:`5601`, :gh:`5611`).
+
+ .. _byte order mark: https://en.wikipedia.org/wiki/Byte_order_mark
+ .. _HTML living standard: https://html.spec.whatwg.org/multipage/parsing.html#determining-the-character-encoding
+
+- MIME sniffing takes the response body into account in FTP and HTTP/1.0
+ requests, as well as in cached requests (:gh:`4873`).
+
+- MIME sniffing now detects valid HTML 5 documents even if the ``html`` tag
+ is missing (:gh:`4873`).
+
+- An exception is now raised if :setting:`ASYNCIO_EVENT_LOOP` has a value
+ that does not match the asyncio event loop actually installed
+ (:gh:`5529`).
+
+- Fixed :meth:`Headers.getlist() <scrapy.http.headers.Headers.getlist>`
+ returning only the last header (:gh:`5515`, :gh:`5526`).
+
+- Fixed :class:`LinkExtractor
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` not ignoring the
+ ``tar.gz`` file extension by default (:gh:`1837`, :gh:`2067`,
+ :gh:`4066`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+- Clarified the return type of :meth:`Spider.parse <scrapy.Spider.parse>`
+ (:gh:`5602`, :gh:`5608`).
+
+- To enable
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ to do `brotli compression`_, installing brotli_ is now recommended instead
+ of installing brotlipy_, as the former provides a more recent version of
+ brotli.
+
+ .. _brotli: https://github.com/google/brotli
+ .. _brotli compression: https://www.ietf.org/rfc/rfc7932.txt
+
+- :ref:`Signal documentation <topics-signals>` now mentions :ref:`coroutine
+ support <topics-coroutines>` and uses it in code examples (:gh:`4852`,
+ :gh:`5358`).
+
+- :ref:`bans` now recommends `Common Crawl`_ instead of `Google cache`_
+ (:gh:`3582`, :gh:`5432`).
+
+ .. _Common Crawl: https://commoncrawl.org/
+ .. _Google cache: https://www.googleguide.com/cached_pages.html
+
+- The new :ref:`topics-components` topic covers enforcing requirements on
+ Scrapy components, like :ref:`downloader middlewares
+ <topics-downloader-middleware>`, :ref:`extensions <topics-extensions>`,
+ :ref:`item pipelines <topics-item-pipeline>`, :ref:`spider middlewares
+ <topics-spider-middleware>`, and more; :ref:`enforce-asyncio-requirement`
+ has also been added (:gh:`4978`).
+
+- :ref:`topics-settings` now indicates that setting values must be
+ :ref:`picklable <pickle-picklable>` (:gh:`5607`, :gh:`5629`).
+
+- Removed outdated documentation (:gh:`5446`, :gh:`5373`,
+ :gh:`5369`, :gh:`5370`, :gh:`5554`).
+
+- Fixed typos (:gh:`5442`, :gh:`5455`, :gh:`5457`, :gh:`5461`,
+ :gh:`5538`, :gh:`5553`, :gh:`5558`, :gh:`5624`, :gh:`5631`).
+
+- Fixed other issues (:gh:`5283`, :gh:`5284`, :gh:`5559`,
+ :gh:`5567`, :gh:`5648`, :gh:`5659`, :gh:`5665`).
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+- Added a continuous integration job to run `twine check`_ (:gh:`5655`,
+ :gh:`5656`).
+
+ .. _twine check: https://twine.readthedocs.io/en/stable/#twine-check
+
+- Addressed test issues and warnings (:gh:`5560`, :gh:`5561`,
+ :gh:`5612`, :gh:`5617`, :gh:`5639`, :gh:`5645`, :gh:`5662`,
+ :gh:`5671`, :gh:`5675`).
+
+- Cleaned up code (:gh:`4991`, :gh:`4995`, :gh:`5451`,
+ :gh:`5487`, :gh:`5542`, :gh:`5667`, :gh:`5668`, :gh:`5672`).
+
+- Applied minor code improvements (:gh:`5661`).
+
+
+.. _release-2.6.3:
+
+Scrapy 2.6.3 (2022-09-27)
+-------------------------
+
+- Added support for pyOpenSSL_ 22.1.0, removing support for SSLv3
+ (:gh:`5634`, :gh:`5635`, :gh:`5636`).
+
+- Upgraded the minimum versions of the following dependencies:
+
+ - cryptography_: 2.0 → 3.3
+
+ - pyOpenSSL_: 16.2.0 → 21.0.0
+
+ - service_identity_: 16.0.0 → 18.1.0
+
+ - Twisted_: 17.9.0 → 18.9.0
+
+ - zope.interface_: 4.1.3 → 5.0.0
+
+ (:gh:`5621`, :gh:`5632`)
+
+- Fixes test and documentation issues (:gh:`5612`, :gh:`5617`,
+ :gh:`5631`).
+
+
+.. _release-2.6.2:
+
+Scrapy 2.6.2 (2022-07-25)
+-------------------------
+
+**Security bug fix:**
+
+- When :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
+ processes a request with :reqmeta:`proxy` metadata, and that
+ :reqmeta:`proxy` metadata includes proxy credentials,
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` sets
+ the ``Proxy-Authorization`` header, but only if that header is not already
+ set.
+
+ There are third-party proxy-rotation downloader middlewares that set
+ different :reqmeta:`proxy` metadata every time they process a request.
+
+ Because of request retries and redirects, the same request can be processed
+ by downloader middlewares more than once, including both
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and
+ any third-party proxy-rotation downloader middleware.
+
+ These third-party proxy-rotation downloader middlewares could change the
+ :reqmeta:`proxy` metadata of a request to a new value, but fail to remove
+ the ``Proxy-Authorization`` header from the previous value of the
+ :reqmeta:`proxy` metadata, causing the credentials of one proxy to be sent
+ to a different proxy.
+
+ To prevent the unintended leaking of proxy credentials, the behavior of
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` is now
+ as follows when processing a request:
+
+ - If the request being processed defines :reqmeta:`proxy` metadata that
+ includes credentials, the ``Proxy-Authorization`` header is always
+ updated to feature those credentials.
+
+ - If the request being processed defines :reqmeta:`proxy` metadata
+ without credentials, the ``Proxy-Authorization`` header is removed
+ *unless* it was originally defined for the same proxy URL.
+
+ To remove proxy credentials while keeping the same proxy URL, remove
+ the ``Proxy-Authorization`` header.
+
+ - If the request has no :reqmeta:`proxy` metadata, or that metadata is a
+ falsy value (e.g. ``None``), the ``Proxy-Authorization`` header is
+ removed.
+
+ It is no longer possible to set a proxy URL through the
+ :reqmeta:`proxy` metadata but set the credentials through the
+ ``Proxy-Authorization`` header. Set proxy credentials through the
+ :reqmeta:`proxy` metadata instead.
+
+Also fixes the following regressions introduced in 2.6.0:
+
+- :class:`~scrapy.crawler.CrawlerProcess` supports again crawling multiple
+ spiders (:gh:`5435`, :gh:`5436`)
+
+- Installing a Twisted reactor before Scrapy does (e.g. importing
+ :mod:`twisted.internet.reactor` somewhere at the module level) no longer
+ prevents Scrapy from starting, as long as a different reactor is not
+ specified in :setting:`TWISTED_REACTOR` (:gh:`5525`, :gh:`5528`)
+
+- Fixed an exception that was being logged after the spider finished under
+ certain conditions (:gh:`5437`, :gh:`5440`)
+
+- The ``--output``/``-o`` command-line parameter supports again a value
+ starting with a hyphen (:gh:`5444`, :gh:`5445`)
+
+- The ``scrapy parse -h`` command no longer throws an error (:gh:`5481`,
+ :gh:`5482`)
+
+
+.. _release-2.6.1:
+
+Scrapy 2.6.1 (2022-03-01)
+-------------------------
+
+Fixes a regression introduced in 2.6.0 that would unset the request method when
+following redirects.
+
+
+.. _release-2.6.0:
+
+Scrapy 2.6.0 (2022-03-01)
+-------------------------
+
+Highlights:
+
+* :ref:`Security fixes for cookie handling <2.6-security-fixes>`
+
+* Python 3.10 support
+
+* :ref:`asyncio support <using-asyncio>` is no longer considered
+ experimental, and works out-of-the-box on Windows regardless of your Python
+ version
+
+* Feed exports now support :class:`pathlib.Path` output paths and per-feed
+ :ref:`item filtering <item-filter>` and
+ :ref:`post-processing <post-processing>`
+
+.. _2.6-security-fixes:
+
+Security bug fixes
+~~~~~~~~~~~~~~~~~~
+
+- When a :class:`~scrapy.Request` object with cookies defined gets a
+ redirect response causing a new :class:`~scrapy.Request` object to be
+ scheduled, the cookies defined in the original
+ :class:`~scrapy.Request` object are no longer copied into the new
+ :class:`~scrapy.Request` object.
+
+ If you manually set the ``Cookie`` header on a
+ :class:`~scrapy.Request` object and the domain name of the redirect
+ URL is not an exact match for the domain of the URL of the original
+ :class:`~scrapy.Request` object, your ``Cookie`` header is now dropped
+ from the new :class:`~scrapy.Request` object.
+
+ The old behavior could be exploited by an attacker to gain access to your
+ cookies. Please, see the `cjvr-mfj7-j4j8 security advisory`_ for more
+ information.
+
+ .. _cjvr-mfj7-j4j8 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cjvr-mfj7-j4j8
+
+ .. note:: It is still possible to enable the sharing of cookies between
+ different domains with a shared domain suffix (e.g.
+ ``example.com`` and any subdomain) by defining the shared domain
+ suffix (e.g. ``example.com``) as the cookie domain when defining
+ your cookies. See the documentation of the
+ :class:`~scrapy.Request` class for more information.
+
+- When the domain of a cookie, either received in the ``Set-Cookie`` header
+ of a response or defined in a :class:`~scrapy.Request` object, is set
+ to a `public suffix <https://publicsuffix.org/>`_, the cookie is now
+ ignored unless the cookie domain is the same as the request domain.
+
+ The old behavior could be exploited by an attacker to inject cookies from a
+ controlled domain into your cookiejar that could be sent to other domains
+ not controlled by the attacker. Please, see the `mfjm-vh54-3f96 security
+ advisory`_ for more information.
+
+ .. _mfjm-vh54-3f96 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-mfjm-vh54-3f96
+
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+- The h2_ dependency is now optional, only needed to
+ :ref:`enable HTTP/2 support <twisted-http2-handler>`. (:gh:`5113`)
+
+ .. _h2: https://pypi.org/project/h2/
+
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- The ``formdata`` parameter of :class:`~scrapy.FormRequest`, if specified
+ for a non-POST request, now overrides the URL query string, instead of
+ being appended to it. (:gh:`2919`, :gh:`3579`)
+
+- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting, now
+ the return value of that function, and not the ``params`` input parameter,
+ will determine the feed URI parameters, unless that return value is
+ ``None``. (:gh:`4962`, :gh:`4966`)
+
+- In :class:`scrapy.core.engine.ExecutionEngine`, methods
+ :meth:`~scrapy.core.engine.ExecutionEngine.crawl`,
+ :meth:`~scrapy.core.engine.ExecutionEngine.download`,
+ :meth:`~scrapy.core.engine.ExecutionEngine.schedule`,
+ and :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
+ now raise :exc:`RuntimeError` if called before
+ :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`. (:gh:`5090`)
+
+ These methods used to assume that
+ :attr:`ExecutionEngine.slot <scrapy.core.engine.ExecutionEngine.slot>` had
+ been defined by a prior call to
+ :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`, so they were
+ raising :exc:`AttributeError` instead.
+
+- If the API of the configured :ref:`scheduler <topics-scheduler>` does not
+ meet expectations, :exc:`TypeError` is now raised at startup time. Before,
+ other exceptions would be raised at run time. (:gh:`3559`)
+
+- The ``_encoding`` field of serialized :class:`~scrapy.Request` objects
+ is now named ``encoding``, in line with all other fields (:gh:`5130`)
+
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- ``scrapy.http.TextResponse.body_as_unicode``, deprecated in Scrapy 2.2, has
+ now been removed. (:gh:`5393`)
+
+- ``scrapy.item.BaseItem``, deprecated in Scrapy 2.2, has now been removed.
+ (:gh:`5398`)
+
+- ``scrapy.item.DictItem``, deprecated in Scrapy 1.8, has now been removed.
+ (:gh:`5398`)
+
+- ``scrapy.Spider.make_requests_from_url``, deprecated in Scrapy 1.4, has now
+ been removed. (:gh:`4178`, :gh:`4356`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+- When a function is assigned to the :setting:`FEED_URI_PARAMS` setting,
+ returning ``None`` or modifying the ``params`` input parameter is now
+ deprecated. Return a new dictionary instead. (:gh:`4962`, :gh:`4966`)
+
+- :mod:`scrapy.utils.reqser` is deprecated. (:gh:`5130`)
+
+ - Instead of :func:`~scrapy.utils.reqser.request_to_dict`, use the new
+ :meth:`.Request.to_dict` method.
+
+ - Instead of :func:`~scrapy.utils.reqser.request_from_dict`, use the new
+ :func:`scrapy.utils.request.request_from_dict` function.
+
+- In :mod:`scrapy.squeues`, the following queue classes are deprecated:
+ :class:`~scrapy.squeues.PickleFifoDiskQueueNonRequest`,
+ :class:`~scrapy.squeues.PickleLifoDiskQueueNonRequest`,
+ :class:`~scrapy.squeues.MarshalFifoDiskQueueNonRequest`,
+ and :class:`~scrapy.squeues.MarshalLifoDiskQueueNonRequest`. You should
+ instead use:
+ :class:`~scrapy.squeues.PickleFifoDiskQueue`,
+ :class:`~scrapy.squeues.PickleLifoDiskQueue`,
+ :class:`~scrapy.squeues.MarshalFifoDiskQueue`,
+ and :class:`~scrapy.squeues.MarshalLifoDiskQueue`. (:gh:`5117`)
+
+- Many aspects of :class:`scrapy.core.engine.ExecutionEngine` that come from
+ a time when this class could handle multiple :class:`~scrapy.Spider`
+ objects at a time have been deprecated. (:gh:`5090`)
+
+ - The :meth:`~scrapy.core.engine.ExecutionEngine.has_capacity` method
+ is deprecated.
+
+ - The :meth:`~scrapy.core.engine.ExecutionEngine.schedule` method is
+ deprecated, use :meth:`~scrapy.core.engine.ExecutionEngine.crawl` or
+ :meth:`~scrapy.core.engine.ExecutionEngine.download` instead.
+
+ - The :attr:`~scrapy.core.engine.ExecutionEngine.open_spiders` attribute
+ is deprecated, use :attr:`~scrapy.core.engine.ExecutionEngine.spider`
+ instead.
+
+ - The ``spider`` parameter is deprecated for the following methods:
+
+ - :meth:`~scrapy.core.engine.ExecutionEngine.spider_is_idle`
+
+ - :meth:`~scrapy.core.engine.ExecutionEngine.crawl`
+
+ - :meth:`~scrapy.core.engine.ExecutionEngine.download`
+
+ Instead, call :meth:`~scrapy.core.engine.ExecutionEngine.open_spider`
+ first to set the :class:`~scrapy.Spider` object.
+
+- :func:`scrapy.utils.response.response_httprepr` is now deprecated.
+ (:gh:`4972`)
+
+
+New features
+~~~~~~~~~~~~
+
+- You can now use :ref:`item filtering <item-filter>` to control which items
+ are exported to each output feed. (:gh:`4575`, :gh:`5178`,
+ :gh:`5161`, :gh:`5203`)
+
+- You can now apply :ref:`post-processing <post-processing>` to feeds, and
+ :ref:`built-in post-processing plugins <builtin-plugins>` are provided for
+ output file compression. (:gh:`2174`, :gh:`5168`, :gh:`5190`)
+
+- The :setting:`FEEDS` setting now supports :class:`pathlib.Path` objects as
+ keys. (:gh:`5383`, :gh:`5384`)
+
+- Enabling :ref:`asyncio <using-asyncio>` while using Windows and Python 3.8
+ or later will automatically switch the asyncio event loop to one that
+ allows Scrapy to work. See :ref:`asyncio-windows`. (:gh:`4976`,
+ :gh:`5315`)
+
+- The :command:`genspider` command now supports a start URL instead of a
+ domain name. (:gh:`4439`)
+
+- :mod:`scrapy.utils.defer` gained 2 new functions,
+ :func:`~scrapy.utils.defer.deferred_to_future` and
+ :func:`~scrapy.utils.defer.maybe_deferred_to_future`, to help :ref:`await
+ on Deferreds when using the asyncio reactor <asyncio-await-dfd>`.
+ (:gh:`5288`)
+
+- :ref:`Amazon S3 feed export storage <topics-feed-storage-s3>` gained
+ support for `temporary security credentials`_
+ (:setting:`AWS_SESSION_TOKEN`) and endpoint customization
+ (:setting:`AWS_ENDPOINT_URL`). (:gh:`4998`, :gh:`5210`)
+
+ .. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
+
+- New :setting:`LOG_FILE_APPEND` setting to allow truncating the log file.
+ (:gh:`5279`)
+
+- :attr:`Request.cookies <scrapy.Request.cookies>` values that are
+ :class:`bool`, :class:`float` or :class:`int` are cast to :class:`str`.
+ (:gh:`5252`, :gh:`5253`)
+
+- You may now raise :exc:`~scrapy.exceptions.CloseSpider` from a handler of
+ the :signal:`spider_idle` signal to customize the reason why the spider is
+ stopping. (:gh:`5191`)
+
+- When using
+ :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`, the
+ proxy URL for non-HTTPS HTTP/1.1 requests no longer needs to include a URL
+ scheme. (:gh:`4505`, :gh:`4649`)
+
+- All built-in queues now expose a ``peek`` method that returns the next
+ queue object (like ``pop``) but does not remove the returned object from
+ the queue. (:gh:`5112`)
+
+ If the underlying queue does not support peeking (e.g. because you are not
+ using ``queuelib`` 1.6.1 or later), the ``peek`` method raises
+ :exc:`NotImplementedError`.
+
+- :class:`~scrapy.Request` and :class:`~scrapy.http.Response` now have
+ an ``attributes`` attribute that makes subclassing easier. For
+ :class:`~scrapy.Request`, it also allows subclasses to work with
+ :func:`scrapy.utils.request.request_from_dict`. (:gh:`1877`,
+ :gh:`5130`, :gh:`5218`)
+
+- The :meth:`~scrapy.core.scheduler.BaseScheduler.open` and
+ :meth:`~scrapy.core.scheduler.BaseScheduler.close` methods of the
+ :ref:`scheduler <topics-scheduler>` are now optional. (:gh:`3559`)
+
+- HTTP/1.1 :exc:`~scrapy.core.downloader.handlers.http11.TunnelError`
+ exceptions now only truncate response bodies longer than 1000 characters,
+ instead of those longer than 32 characters, making it easier to debug such
+ errors. (:gh:`4881`, :gh:`5007`)
+
+- :class:`~scrapy.loader.ItemLoader` now supports non-text responses.
+ (:gh:`5145`, :gh:`5269`)
+
+
+Bug fixes
+~~~~~~~~~
+
+- The :setting:`TWISTED_REACTOR` and :setting:`ASYNCIO_EVENT_LOOP` settings
+ are no longer ignored if defined in :attr:`~scrapy.Spider.custom_settings`.
+ (:gh:`4485`, :gh:`5352`)
+
+- Removed a module-level Twisted reactor import that could prevent
+ :ref:`using the asyncio reactor <using-asyncio>`. (:gh:`5357`)
+
+- The :command:`startproject` command works with existing folders again.
+ (:gh:`4665`, :gh:`4676`)
+
+- The :setting:`FEED_URI_PARAMS` setting now behaves as documented.
+ (:gh:`4962`, :gh:`4966`)
+
+- :attr:`Request.cb_kwargs <scrapy.Request.cb_kwargs>` once again allows the
+ ``callback`` keyword. (:gh:`5237`, :gh:`5251`, :gh:`5264`)
+
+- Made :func:`scrapy.utils.response.open_in_browser` support more complex
+ HTML. (:gh:`5319`, :gh:`5320`)
+
+- Fixed :attr:`CSVFeedSpider.quotechar
+ <scrapy.spiders.CSVFeedSpider.quotechar>` being interpreted as the CSV file
+ encoding. (:gh:`5391`, :gh:`5394`)
+
+- Added missing setuptools_ to the list of dependencies. (:gh:`5122`)
+
+ .. _setuptools: https://pypi.org/project/setuptools/
+
+- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ now also works as expected with links that have comma-separated ``rel``
+ attribute values including ``nofollow``. (:gh:`5225`)
+
+- Fixed a :exc:`TypeError` that could be raised during :ref:`feed export
+ <topics-feed-exports>` parameter parsing. (:gh:`5359`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+- :ref:`asyncio support <using-asyncio>` is no longer considered
+ experimental. (:gh:`5332`)
+
+- Included :ref:`Windows-specific help for asyncio usage <asyncio-windows>`.
+ (:gh:`4976`, :gh:`5315`)
+
+- Rewrote :ref:`topics-headless-browsing` with up-to-date best practices.
+ (:gh:`4484`, :gh:`4613`)
+
+- Documented :ref:`local file naming in media pipelines
+ <topics-file-naming>`. (:gh:`5069`, :gh:`5152`)
+
+- :ref:`faq` now covers spider file name collision issues. (:gh:`2680`,
+ :gh:`3669`)
+
+- Provided better context and instructions to disable the
+ :setting:`URLLENGTH_LIMIT` setting. (:gh:`5135`, :gh:`5250`)
+
+- Documented that Reppy parser does not support Python 3.9+.
+ (:gh:`5226`, :gh:`5231`)
+
+- Documented :ref:`the scheduler component <topics-scheduler>`.
+ (:gh:`3537`, :gh:`3559`)
+
+- Documented the method used by :ref:`media pipelines
+ <topics-media-pipeline>` to :ref:`determine if a file has expired
+ <file-expiration>`. (:gh:`5120`, :gh:`5254`)
+
+- :ref:`run-multiple-spiders` now features
+ :func:`scrapy.utils.project.get_project_settings` usage. (:gh:`5070`)
+
+- :ref:`run-multiple-spiders` now covers what happens when you define
+ different per-spider values for some settings that cannot differ at run
+ time. (:gh:`4485`, :gh:`5352`)
+
+- Extended the documentation of the
+ :class:`~scrapy.extensions.statsmailer.StatsMailer` extension.
+ (:gh:`5199`, :gh:`5217`)
+
+- Added :setting:`JOBDIR` to :ref:`topics-settings`. (:gh:`5173`,
+ :gh:`5224`)
+
+- Documented :attr:`Spider.attribute <scrapy.Spider.attribute>`.
+ (:gh:`5174`, :gh:`5244`)
+
+- Documented :attr:`TextResponse.urljoin <scrapy.http.TextResponse.urljoin>`.
+ (:gh:`1582`)
+
+- Added the ``body_length`` parameter to the documented signature of the
+ :signal:`headers_received` signal. (:gh:`5270`)
+
+- Clarified :meth:`SelectorList.get <scrapy.selector.SelectorList.get>` usage
+ in the :ref:`tutorial <intro-tutorial>`. (:gh:`5256`)
+
+- The documentation now features the shortest import path of classes with
+ multiple import paths. (:gh:`2733`, :gh:`5099`)
+
+- ``quotes.toscrape.com`` references now use HTTPS instead of HTTP.
+ (:gh:`5395`, :gh:`5396`)
+
+- Added a link to `our Discord server <https://discord.com/invite/mv3yErfpvq>`_
+ to :ref:`getting-help`. (:gh:`5421`, :gh:`5422`)
+
+- The pronunciation of the project name is now :ref:`officially
+ <intro-overview>` /ˈskreɪpaɪ/. (:gh:`5280`, :gh:`5281`)
+
+- Added the Scrapy logo to the README. (:gh:`5255`, :gh:`5258`)
+
+- Fixed issues and implemented minor improvements. (:gh:`3155`,
+ :gh:`4335`, :gh:`5074`, :gh:`5098`, :gh:`5134`, :gh:`5180`,
+ :gh:`5194`, :gh:`5239`, :gh:`5266`, :gh:`5271`, :gh:`5273`,
+ :gh:`5274`, :gh:`5276`, :gh:`5347`, :gh:`5356`, :gh:`5414`,
+ :gh:`5415`, :gh:`5416`, :gh:`5419`, :gh:`5420`)
+
+
+Quality Assurance
+~~~~~~~~~~~~~~~~~
+
+- Added support for Python 3.10. (:gh:`5212`, :gh:`5221`,
+ :gh:`5265`)
+
+- Significantly reduced memory usage by
+ :func:`scrapy.utils.response.response_httprepr`, used by the
+ :class:`~scrapy.downloadermiddlewares.stats.DownloaderStats` downloader
+ middleware, which is enabled by default. (:gh:`4964`, :gh:`4972`)
+
+- Removed uses of the deprecated :mod:`optparse` module. (:gh:`5366`,
+ :gh:`5374`)
+
+- Extended typing hints. (:gh:`5077`, :gh:`5090`, :gh:`5100`,
+ :gh:`5108`, :gh:`5171`, :gh:`5215`, :gh:`5334`)
+
+- Improved tests, fixed CI issues, removed unused code. (:gh:`5094`,
+ :gh:`5157`, :gh:`5162`, :gh:`5198`, :gh:`5207`, :gh:`5208`,
+ :gh:`5229`, :gh:`5298`, :gh:`5299`, :gh:`5310`, :gh:`5316`,
+ :gh:`5333`, :gh:`5388`, :gh:`5389`, :gh:`5400`, :gh:`5401`,
+ :gh:`5404`, :gh:`5405`, :gh:`5407`, :gh:`5410`, :gh:`5412`,
+ :gh:`5425`, :gh:`5427`)
+
+- Implemented improvements for contributors. (:gh:`5080`, :gh:`5082`,
+ :gh:`5177`, :gh:`5200`)
+
+- Implemented cleanups. (:gh:`5095`, :gh:`5106`, :gh:`5209`,
+ :gh:`5228`, :gh:`5235`, :gh:`5245`, :gh:`5246`, :gh:`5292`,
+ :gh:`5314`, :gh:`5322`)
+
+
+.. _release-2.5.1:
+
+Scrapy 2.5.1 (2021-10-05)
+-------------------------
+
+* **Security bug fix:**
+
+ If you use
+ :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
+ (i.e. the ``http_user`` and ``http_pass`` spider attributes) for HTTP
+ authentication, any request exposes your credentials to the request target.
+
+ To prevent unintended exposure of authentication credentials to unintended
+ domains, you must now additionally set a new, additional spider attribute,
+ ``http_auth_domain``, and point it to the specific domain to which the
+ authentication credentials must be sent.
+
+ If the ``http_auth_domain`` spider attribute is not set, the domain of the
+ first request will be considered the HTTP authentication target, and
+ authentication credentials will only be sent in requests targeting that
+ domain.
+
+ If you need to send the same HTTP authentication credentials to multiple
+ domains, you can use :func:`w3lib.http.basic_auth_header` instead to
+ set the value of the ``Authorization`` header of your requests.
+
+ If you *really* want your spider to send the same HTTP authentication
+ credentials to any domain, set the ``http_auth_domain`` spider attribute
+ to ``None``.
+
+ Finally, if you are a user of `scrapy-splash`_, know that this version of
+ Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will
+ need to upgrade scrapy-splash to a greater version for it to continue to
+ work.
+
+
+.. _release-2.5.0:
+
+Scrapy 2.5.0 (2021-04-06)
+-------------------------
+
+Highlights:
+
+- Official Python 3.9 support
+
+- Experimental :ref:`HTTP/2 support <twisted-http2-handler>`
+
+- New :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function
+ to retry requests from spider callbacks
+
+- New :class:`~scrapy.signals.headers_received` signal that allows stopping
+ downloads early
+
+- New :class:`Response.protocol <scrapy.http.Response.protocol>` attribute
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+- Removed all code that :ref:`was deprecated in 1.7.0 <1.7-deprecations>` and
+ had not :ref:`already been removed in 2.4.0 <2.4-deprecation-removals>`.
+ (:gh:`4901`)
+
+- Removed support for the ``SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE`` environment
+ variable, :ref:`deprecated in 1.8.0 <1.8-deprecations>`. (:gh:`4912`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+- The :mod:`scrapy.utils.py36` module is now deprecated in favor of
+ :mod:`scrapy.utils.asyncgen`. (:gh:`4900`)
+
+
+New features
+~~~~~~~~~~~~
+
+- Experimental :ref:`HTTP/2 support <twisted-http2-handler>` through a new download handler
+ that can be assigned to the ``https`` protocol in the
+ :setting:`DOWNLOAD_HANDLERS` setting.
+ (:gh:`1854`, :gh:`4769`, :gh:`5058`, :gh:`5059`, :gh:`5066`)
+
+- The new :func:`scrapy.downloadermiddlewares.retry.get_retry_request`
+ function may be used from spider callbacks or middlewares to handle the
+ retrying of a request beyond the scenarios that
+ :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` supports.
+ (:gh:`3590`, :gh:`3685`, :gh:`4902`)
+
+- The new :class:`~scrapy.signals.headers_received` signal gives early access
+ to response headers and allows :ref:`stopping downloads
+ <topics-stop-response-download>`.
+ (:gh:`1772`, :gh:`4897`)
+
+- The new :attr:`Response.protocol <scrapy.http.Response.protocol>`
+ attribute gives access to the string that identifies the protocol used to
+ download a response. (:gh:`4878`)
+
+- :ref:`Stats <topics-stats>` now include the following entries that indicate
+ the number of successes and failures in storing
+ :ref:`feeds <topics-feed-exports>`::
+
+ feedexport/success_count/<storage type>
+ feedexport/failed_count/<storage type>
+
+ Where ``<storage type>`` is the feed storage backend class name, such as
+ :class:`~scrapy.extensions.feedexport.FileFeedStorage` or
+ :class:`~scrapy.extensions.feedexport.FTPFeedStorage`.
+
+ (:gh:`3947`, :gh:`4850`)
+
+- The :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` spider
+ middleware now logs ignored URLs with ``INFO`` :ref:`logging level
+ <levels>` instead of ``DEBUG``, and it now includes the following entry
+ into :ref:`stats <topics-stats>` to keep track of the number of ignored
+ URLs::
+
+ urllength/request_ignored_count
+
+ (:gh:`5036`)
+
+- The
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ downloader middleware now logs the number of decompressed responses and the
+ total count of resulting bytes::
+
+ httpcompression/response_bytes
+ httpcompression/response_count
+
+ (:gh:`4797`, :gh:`4799`)
+
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed installation on PyPy installing PyDispatcher in addition to
+ PyPyDispatcher, which could prevent Scrapy from working depending on which
+ package got imported. (:gh:`4710`, :gh:`4814`)
+
+- When inspecting a callback to check if it is a generator that also returns
+ a value, an exception is no longer raised if the callback has a docstring
+ with lower indentation than the following code.
+ (:gh:`4477`, :gh:`4935`)
+
+- The `Content-Length <https://datatracker.ietf.org/doc/html/rfc2616#section-14.13>`_
+ header is no longer omitted from responses when using the default, HTTP/1.1
+ download handler (see :setting:`DOWNLOAD_HANDLERS`).
+ (:gh:`5009`, :gh:`5034`, :gh:`5045`, :gh:`5057`, :gh:`5062`)
+
+- Setting the :reqmeta:`handle_httpstatus_all` request meta key to ``False``
+ now has the same effect as not setting it at all, instead of having the
+ same effect as setting it to ``True``.
+ (:gh:`3851`, :gh:`4694`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+- Added instructions to :ref:`install Scrapy in Windows using pip
+ <intro-install-windows>`.
+ (:gh:`4715`, :gh:`4736`)
+
+- Logging documentation now includes :ref:`additional ways to filter logs
+ <topics-logging-advanced-customization>`.
+ (:gh:`4216`, :gh:`4257`, :gh:`4965`)
+
+- Covered how to deal with long lists of allowed domains in the :ref:`FAQ
+ <faq>`. (:gh:`2263`, :gh:`3667`)
+
+- Covered scrapy-bench_ in :ref:`benchmarking`.
+ (:gh:`4996`, :gh:`5016`)
+
+- Clarified that one :ref:`extension <topics-extensions>` instance is created
+ per crawler.
+ (:gh:`5014`)
+
+- Fixed some errors in examples.
+ (:gh:`4829`, :gh:`4830`, :gh:`4907`, :gh:`4909`,
+ :gh:`5008`)
+
+- Fixed some external links, typos, and so on.
+ (:gh:`4892`, :gh:`4899`, :gh:`4936`, :gh:`4942`, :gh:`5005`,
+ :gh:`5063`)
+
+- The :ref:`list of Request.meta keys <topics-request-meta>` is now sorted
+ alphabetically.
+ (:gh:`5061`, :gh:`5065`)
+
+- Updated references to Scrapinghub, which is now called Zyte.
+ (:gh:`4973`, :gh:`5072`)
+
+- Added a mention to contributors in the README. (:gh:`4956`)
+
+- Reduced the top margin of lists. (:gh:`4974`)
+
+
+Quality Assurance
+~~~~~~~~~~~~~~~~~
+
+- Made Python 3.9 support official (:gh:`4757`, :gh:`4759`)
+
+- Extended typing hints (:gh:`4895`)
+
+- Fixed deprecated uses of the Twisted API.
+ (:gh:`4940`, :gh:`4950`, :gh:`5073`)
+
+- Made our tests run with the new pip resolver.
+ (:gh:`4710`, :gh:`4814`)
+
+- Added tests to ensure that :ref:`coroutine support <coroutine-support>`
+ is tested. (:gh:`4987`)
+
+- Migrated from Travis CI to GitHub Actions. (:gh:`4924`)
+
+- Fixed CI issues.
+ (:gh:`4986`, :gh:`5020`, :gh:`5022`, :gh:`5027`, :gh:`5052`,
+ :gh:`5053`)
+
+- Implemented code refactorings, style fixes and cleanups.
+ (:gh:`4911`, :gh:`4982`, :gh:`5001`, :gh:`5002`, :gh:`5076`)
+
+
+.. _release-2.4.1:
+
+Scrapy 2.4.1 (2020-11-17)
+-------------------------
+
+- Fixed :ref:`feed exports <topics-feed-exports>` overwrite support (:gh:`4845`, :gh:`4857`, :gh:`4859`)
+
+- Fixed the AsyncIO event loop handling, which could make code hang
+ (:gh:`4855`, :gh:`4872`)
+
+- Fixed the IPv6-capable DNS resolver
+ :class:`~scrapy.resolver.CachingHostnameResolver` for download handlers
+ that call
+ :meth:`reactor.resolve <twisted.internet.interfaces.IReactorCore.resolve>`
+ (:gh:`4802`, :gh:`4803`)
+
+- Fixed the output of the :command:`genspider` command showing placeholders
+ instead of the import path of the generated spider module (:gh:`4874`)
+
+- Migrated Windows CI from Azure Pipelines to GitHub Actions (:gh:`4869`,
+ :gh:`4876`)
+
+
+.. _release-2.4.0:
+
+Scrapy 2.4.0 (2020-10-11)
+-------------------------
+
+Highlights:
+
+* Python 3.5 support has been dropped.
+
+* The ``file_path`` method of :ref:`media pipelines <topics-media-pipeline>`
+ can now access the source :ref:`item <topics-items>`.
+
+ This allows you to set a download file path based on item data.
+
+* The new ``item_export_kwargs`` key of the :setting:`FEEDS` setting allows
+ to define keyword parameters to pass to :ref:`item exporter classes
+ <topics-exporters>`
+
+* You can now choose whether :ref:`feed exports <topics-feed-exports>`
+ overwrite or append to the output file.
+
+ For example, when using the :command:`crawl` or :command:`runspider`
+ commands, you can use the ``-O`` option instead of ``-o`` to overwrite the
+ output file.
+
+* Zstd-compressed responses are now supported if zstandard_ is installed.
+
+* In settings, where the import path of a class is required, it is now
+ possible to pass a class object instead.
+
+Modified requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+* Python 3.6 or greater is now required; support for Python 3.5 has been
+ dropped
+
+ As a result:
+
+ - When using PyPy, PyPy 7.2.0 or greater :ref:`is now required
+ <faq-python-versions>`
+
+ - For Amazon S3 storage support in :ref:`feed exports
+ <topics-feed-storage-s3>` or :ref:`media pipelines
+ <media-pipelines-s3>`, botocore_ 1.4.87 or greater is now required
+
+ - To use the :ref:`images pipeline <images-pipeline>`, Pillow_ 4.0.0 or
+ greater is now required
+
+ (:gh:`4718`, :gh:`4732`, :gh:`4733`, :gh:`4742`, :gh:`4743`,
+ :gh:`4764`)
+
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` once again
+ discards cookies defined in :attr:`.Request.headers`.
+
+ We decided to revert this bug fix, introduced in Scrapy 2.2.0, because it
+ was reported that the current implementation could break existing code.
+
+ If you need to set cookies for a request, use the :class:`Request.cookies
+ <scrapy.Request>` parameter.
+
+ A future version of Scrapy will include a new, better implementation of the
+ reverted bug fix.
+
+ (:gh:`4717`, :gh:`4823`)
+
+
+.. _2.4-deprecation-removals:
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* :class:`scrapy.extensions.feedexport.S3FeedStorage` no longer reads the
+ values of ``access_key`` and ``secret_key`` from the running project
+ settings when they are not passed to its ``__init__`` method; you must
+ either pass those parameters to its ``__init__`` method or use
+ :class:`S3FeedStorage.from_crawler
+ <scrapy.extensions.feedexport.S3FeedStorage.from_crawler>`
+ (:gh:`4356`, :gh:`4411`, :gh:`4688`)
+
+* :attr:`Rule.process_request <scrapy.spiders.crawl.Rule.process_request>`
+ no longer admits callables which expect a single ``request`` parameter,
+ rather than both ``request`` and ``response`` (:gh:`4818`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* In custom :ref:`media pipelines <topics-media-pipeline>`, signatures that
+ do not accept a keyword-only ``item`` parameter in any of the methods that
+ :ref:`now support this parameter <media-pipeline-item-parameter>` are now
+ deprecated (:gh:`4628`, :gh:`4686`)
+
+* In custom :ref:`feed storage backend classes <topics-feed-storage>`,
+ ``__init__`` method signatures that do not accept a keyword-only
+ ``feed_options`` parameter are now deprecated (:gh:`547`, :gh:`716`,
+ :gh:`4512`)
+
+* The :class:`scrapy.utils.python.WeakKeyCache` class is now deprecated
+ (:gh:`4684`, :gh:`4701`)
+
+* The :func:`scrapy.utils.boto.is_botocore` function is now deprecated, use
+ :func:`scrapy.utils.boto.is_botocore_available` instead (:gh:`4734`,
+ :gh:`4776`)
+
+
+New features
+~~~~~~~~~~~~
+
+.. _media-pipeline-item-parameter:
+
+* The following methods of :ref:`media pipelines <topics-media-pipeline>` now
+ accept an ``item`` keyword-only parameter containing the source
+ :ref:`item <topics-items>`:
+
+ - In :class:`scrapy.pipelines.files.FilesPipeline`:
+
+ - :meth:`~scrapy.pipelines.files.FilesPipeline.file_downloaded`
+
+ - :meth:`~scrapy.pipelines.files.FilesPipeline.file_path`
+
+ - :meth:`~scrapy.pipelines.files.FilesPipeline.media_downloaded`
+
+ - :meth:`~scrapy.pipelines.files.FilesPipeline.media_to_download`
+
+ - In :class:`scrapy.pipelines.images.ImagesPipeline`:
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.file_downloaded`
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.file_path`
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.get_images`
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.image_downloaded`
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.media_downloaded`
+
+ - :meth:`~scrapy.pipelines.images.ImagesPipeline.media_to_download`
+
+ (:gh:`4628`, :gh:`4686`)
+
+* The new ``item_export_kwargs`` key of the :setting:`FEEDS` setting allows
+ to define keyword parameters to pass to :ref:`item exporter classes
+ <topics-exporters>` (:gh:`4606`, :gh:`4768`)
+
+* :ref:`Feed exports <topics-feed-exports>` gained overwrite support:
+
+ * When using the :command:`crawl` or :command:`runspider` commands, you
+ can use the ``-O`` option instead of ``-o`` to overwrite the output
+ file
+
+ * You can use the ``overwrite`` key in the :setting:`FEEDS` setting to
+ configure whether to overwrite the output file (``True``) or append to
+ its content (``False``)
+
+ * The ``__init__`` and ``from_crawler`` methods of :ref:`feed storage
+ backend classes <topics-feed-storage>` now receive a new keyword-only
+ parameter, ``feed_options``, which is a dictionary of :ref:`feed
+ options <feed-options>`
+
+ (:gh:`547`, :gh:`716`, :gh:`4512`)
+
+* Zstd-compressed responses are now supported if zstandard_ is installed
+ (:gh:`4831`)
+
+* In settings, where the import path of a class is required, it is now
+ possible to pass a class object instead (:gh:`3870`, :gh:`3873`).
+
+ This includes also settings where only part of its value is made of an
+ import path, such as :setting:`DOWNLOADER_MIDDLEWARES` or
+ :setting:`DOWNLOAD_HANDLERS`.
+
+* :ref:`Downloader middlewares <topics-downloader-middleware>` can now
+ override :class:`response.request <scrapy.http.Response.request>`.
+
+ If a :ref:`downloader middleware <topics-downloader-middleware>` returns
+ a :class:`~scrapy.http.Response` object from
+ :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`
+ or
+ :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
+ with a custom :class:`~scrapy.Request` object assigned to
+ :class:`response.request <scrapy.http.Response.request>`:
+
+ - The response is handled by the callback of that custom
+ :class:`~scrapy.Request` object, instead of being handled by the
+ callback of the original :class:`~scrapy.Request` object
+
+ - That custom :class:`~scrapy.Request` object is now sent as the
+ ``request`` argument to the :signal:`response_received` signal, instead
+ of the original :class:`~scrapy.Request` object
+
+ (:gh:`4529`, :gh:`4632`)
+
+* When using the :ref:`FTP feed storage backend <topics-feed-storage-ftp>`:
+
+ - It is now possible to set the new ``overwrite`` :ref:`feed option
+ <feed-options>` to ``False`` to append to an existing file instead of
+ overwriting it
+
+ - The FTP password can now be omitted if it is not necessary
+
+ (:gh:`547`, :gh:`716`, :gh:`4512`)
+
+* The ``__init__`` method of :class:`~scrapy.exporters.CsvItemExporter` now
+ supports an ``errors`` parameter to indicate how to handle encoding errors
+ (:gh:`4755`)
+
+* When :ref:`using asyncio <using-asyncio>`, it is now possible to
+ :ref:`set a custom asyncio loop <using-custom-loops>` (:gh:`4306`,
+ :gh:`4414`)
+
+* Serialized requests (see :ref:`topics-jobs`) now support callbacks that are
+ spider methods that delegate on other callable (:gh:`4756`)
+
+* When a response is larger than :setting:`DOWNLOAD_MAXSIZE`, the logged
+ message is now a warning, instead of an error (:gh:`3874`,
+ :gh:`3886`, :gh:`4752`)
+
+
+Bug fixes
+~~~~~~~~~
+
+* The :command:`genspider` command no longer overwrites existing files
+ unless the ``--force`` option is used (:gh:`4561`, :gh:`4616`,
+ :gh:`4623`)
+
+* Cookies with an empty value are no longer considered invalid cookies
+ (:gh:`4772`)
+
+* The :command:`runspider` command now supports files with the ``.pyw`` file
+ extension (:gh:`4643`, :gh:`4646`)
+
+* The :class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`
+ middleware now simply ignores unsupported proxy values (:gh:`3331`,
+ :gh:`4778`)
+
+* Checks for generator callbacks with a ``return`` statement no longer warn
+ about ``return`` statements in nested functions (:gh:`4720`,
+ :gh:`4721`)
+
+* The system file mode creation mask no longer affects the permissions of
+ files generated using the :command:`startproject` command (:gh:`4722`)
+
+* ``scrapy.utils.iterators.xmliter`` now supports namespaced node names
+ (:gh:`861`, :gh:`4746`)
+
+* :class:`~scrapy.Request` objects can now have ``about:`` URLs, which can
+ work when using a headless browser (:gh:`4835`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* The :setting:`FEED_URI_PARAMS` setting is now documented (:gh:`4671`,
+ :gh:`4724`)
+
+* Improved the documentation of
+ :ref:`link extractors <topics-link-extractors>` with an usage example from
+ a spider callback and reference documentation for the
+ :class:`~scrapy.link.Link` class (:gh:`4751`, :gh:`4775`)
+
+* Clarified the impact of :setting:`CONCURRENT_REQUESTS` when using the
+ :class:`~scrapy.extensions.closespider.CloseSpider` extension
+ (:gh:`4836`)
+
+* Removed references to Python 2’s ``unicode`` type (:gh:`4547`,
+ :gh:`4703`)
+
+* We now have an :ref:`official deprecation policy <deprecation-policy>`
+ (:gh:`4705`)
+
+* Our :ref:`documentation policies <documentation-policies>` now cover usage
+ of Sphinx’s :rst:dir:`versionadded` and :rst:dir:`versionchanged`
+ directives, and we have removed usages referencing Scrapy 1.4.0 and earlier
+ versions (:gh:`3971`, :gh:`4310`)
+
+* Other documentation cleanups (:gh:`4090`, :gh:`4782`, :gh:`4800`,
+ :gh:`4801`, :gh:`4809`, :gh:`4816`, :gh:`4825`)
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* Extended typing hints (:gh:`4243`, :gh:`4691`)
+
+* Added tests for the :command:`check` command (:gh:`4663`)
+
+* Fixed test failures on Debian (:gh:`4726`, :gh:`4727`, :gh:`4735`)
+
+* Improved Windows test coverage (:gh:`4723`)
+
+* Switched to :ref:`formatted string literals <f-strings>` where possible
+ (:gh:`4307`, :gh:`4324`, :gh:`4672`)
+
+* Modernized :func:`super` usage (:gh:`4707`)
+
+* Other code and test cleanups (:gh:`1790`, :gh:`3288`, :gh:`4165`,
+ :gh:`4564`, :gh:`4651`, :gh:`4714`, :gh:`4738`, :gh:`4745`,
+ :gh:`4747`, :gh:`4761`, :gh:`4765`, :gh:`4804`, :gh:`4817`,
+ :gh:`4820`, :gh:`4822`, :gh:`4839`)
+
+
+.. _release-2.3.0:
+
+Scrapy 2.3.0 (2020-08-04)
+-------------------------
+
+Highlights:
+
+* :ref:`Feed exports <topics-feed-exports>` now support :ref:`Google Cloud
+ Storage <topics-feed-storage-gcs>` as a storage backend
+
+* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver
+ output items in batches of up to the specified number of items.
+
+ It also serves as a workaround for :ref:`delayed file delivery
+ <delayed-file-delivery>`, which causes Scrapy to only start item delivery
+ after the crawl has finished when using certain storage backends
+ (:ref:`S3 <topics-feed-storage-s3>`, :ref:`FTP <topics-feed-storage-ftp>`,
+ and now :ref:`GCS <topics-feed-storage-gcs>`).
+
+* The base implementation of :ref:`item loaders <topics-loaders>` has been
+ moved into a separate library, :doc:`itemloaders <itemloaders:index>`,
+ allowing usage from outside Scrapy and a separate release schedule
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* Removed the following classes and their parent modules from
+ ``scrapy.linkextractors``:
+
+ * ``htmlparser.HtmlParserLinkExtractor``
+ * ``regex.RegexLinkExtractor``
+ * ``sgml.BaseSgmlLinkExtractor``
+ * ``sgml.SgmlLinkExtractor``
+
+ Use
+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ instead (:gh:`4356`, :gh:`4679`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated
+ (:gh:`4683`)
+
+
+New features
+~~~~~~~~~~~~
+
+* :ref:`Feed exports <topics-feed-exports>` support :ref:`Google Cloud
+ Storage <topics-feed-storage-gcs>` (:gh:`685`, :gh:`3608`)
+
+* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries
+ (:gh:`4250`, :gh:`4434`)
+
+* The :command:`parse` command now allows specifying an output file
+ (:gh:`4317`, :gh:`4377`)
+
+* :meth:`.Request.from_curl` and
+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
+ ``--data-raw`` (:gh:`4612`)
+
+* A ``parse`` callback may now be used in built-in spider subclasses, such
+ as :class:`~scrapy.spiders.CrawlSpider` (:gh:`712`, :gh:`732`,
+ :gh:`781`, :gh:`4254` )
+
+
+Bug fixes
+~~~~~~~~~
+
+* Fixed the :ref:`CSV exporting <topics-feed-format-csv>` of
+ :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
+ <attrs-items>` (:gh:`4667`, :gh:`4668`)
+
+* :meth:`.Request.from_curl` and
+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
+ method to ``POST`` when a request body is specified and no request method
+ is specified (:gh:`4612`)
+
+* The processing of ANSI escape sequences in enabled in Windows 10.0.14393
+ and later, where it is required for colored output (:gh:`4393`,
+ :gh:`4403`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* Updated the `OpenSSL cipher list format`_ link in the documentation about
+ the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:gh:`4653`)
+
+* Simplified the code example in :ref:`topics-loaders-dataclass`
+ (:gh:`4652`)
+
+.. _OpenSSL cipher list format: https://docs.openssl.org/master/man1/openssl-ciphers/#cipher-list-format
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* The base implementation of :ref:`item loaders <topics-loaders>` has been
+ moved into :doc:`itemloaders <itemloaders:index>` (:gh:`4005`,
+ :gh:`4516`)
+
+* Fixed a silenced error in some scheduler tests (:gh:`4644`,
+ :gh:`4645`)
+
+* Renewed the localhost certificate used for SSL tests (:gh:`4650`)
+
+* Removed cookie-handling code specific to Python 2 (:gh:`4682`)
+
+* Stopped using Python 2 unicode literal syntax (:gh:`4704`)
+
+* Stopped using a backlash for line continuation (:gh:`4673`)
+
+* Removed unneeded entries from the MyPy exception list (:gh:`4690`)
+
+* Automated tests now pass on Windows as part of our continuous integration
+ system (:gh:`4458`)
+
+* Automated tests now pass on the latest PyPy version for supported Python
+ versions in our continuous integration system (:gh:`4504`)
+
+
+.. _release-2.2.1:
+
+Scrapy 2.2.1 (2020-07-17)
+-------------------------
+
+* The :command:`startproject` command no longer makes unintended changes to
+ the permissions of files in the destination folder, such as removing
+ execution permissions (:gh:`4662`, :gh:`4666`)
+
+
+.. _release-2.2.0:
+
+Scrapy 2.2.0 (2020-06-24)
+-------------------------
+
+Highlights:
+
+* Python 3.5.2+ is required now
+* :ref:`dataclass objects <dataclass-items>` and
+ :ref:`attrs objects <attrs-items>` are now valid :ref:`item types
+ <item-types>`
+* New :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method
+* New :signal:`bytes_received` signal that allows canceling response download
+* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` fixes
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Support for Python 3.5.0 and 3.5.1 has been dropped; Scrapy now refuses to
+ run with a Python version lower than 3.5.2, which introduced
+ :class:`typing.Type` (:gh:`4615`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* ``TextResponse.body_as_unicode()`` is now deprecated, use
+ :attr:`TextResponse.text <scrapy.http.TextResponse.text>` instead
+ (:gh:`4546`, :gh:`4555`, :gh:`4579`)
+
+* :class:`scrapy.item.BaseItem` is now deprecated, use
+ :class:`scrapy.item.Item` instead (:gh:`4534`)
+
+
+New features
+~~~~~~~~~~~~
+
+* :ref:`dataclass objects <dataclass-items>` and
+ :ref:`attrs objects <attrs-items>` are now valid :ref:`item types
+ <item-types>`, and a new itemadapter_ library makes it easy to
+ write code that :ref:`supports any item type <supporting-item-types>`
+ (:gh:`2749`, :gh:`2807`, :gh:`3761`, :gh:`3881`, :gh:`4642`)
+
+* A new :meth:`TextResponse.json <scrapy.http.TextResponse.json>` method
+ allows to deserialize JSON responses (:gh:`2444`, :gh:`4460`,
+ :gh:`4574`)
+
+* A new :signal:`bytes_received` signal allows monitoring response download
+ progress and :ref:`stopping downloads <topics-stop-response-download>`
+ (:gh:`4205`, :gh:`4559`)
+
+* The dictionaries in the result list of a :ref:`media pipeline
+ <topics-media-pipeline>` now include a new key, ``status``, which indicates
+ if the file was downloaded or, if the file was not downloaded, why it was
+ not downloaded; see :meth:`FilesPipeline.get_media_requests
+ <scrapy.pipelines.files.FilesPipeline.get_media_requests>` for more
+ information (:gh:`2893`, :gh:`4486`)
+
+* When using :ref:`Google Cloud Storage <media-pipeline-gcs>` for
+ a :ref:`media pipeline <topics-media-pipeline>`, a warning is now logged if
+ the configured credentials do not grant the required permissions
+ (:gh:`4346`, :gh:`4508`)
+
+* :ref:`Link extractors <topics-link-extractors>` are now serializable,
+ as long as you do not use :ref:`lambdas <lambda>` for parameters; for
+ example, you can now pass link extractors in :attr:`.Request.cb_kwargs`
+ or :attr:`.Request.meta` when :ref:`persisting
+ scheduled requests <topics-jobs>` (:gh:`4554`)
+
+* Upgraded the :ref:`pickle protocol <pickle-protocols>` that Scrapy uses
+ from protocol 2 to protocol 4, improving serialization capabilities and
+ performance (:gh:`4135`, :gh:`4541`)
+
+* :func:`scrapy.utils.misc.create_instance` now raises a :exc:`TypeError`
+ exception if the resulting instance is ``None`` (:gh:`4528`,
+ :gh:`4532`)
+
+.. _itemadapter: https://github.com/scrapy/itemadapter
+
+
+Bug fixes
+~~~~~~~~~
+
+* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
+ discards cookies defined in :attr:`Request.headers
+ <scrapy.Request.headers>` (:gh:`1992`, :gh:`2400`)
+
+* :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware` no longer
+ re-encodes cookies defined as :class:`bytes` in the ``cookies`` parameter
+ of the ``__init__`` method of :class:`~scrapy.Request`
+ (:gh:`2400`, :gh:`3575`)
+
+* When :setting:`FEEDS` defines multiple URIs, :setting:`FEED_STORE_EMPTY` is
+ ``False`` and the crawl yields no items, Scrapy no longer stops feed
+ exports after the first URI (:gh:`4621`, :gh:`4626`)
+
+* :class:`~scrapy.spiders.Spider` callbacks defined using :doc:`coroutine
+ syntax <topics/coroutines>` no longer need to return an iterable, and may
+ instead return a :class:`~scrapy.Request` object, an
+ :ref:`item <topics-items>`, or ``None`` (:gh:`4609`)
+
+* The :command:`startproject` command now ensures that the generated project
+ folders and files have the right permissions (:gh:`4604`)
+
+* Fix a :exc:`KeyError` exception being sometimes raised from
+ :class:`scrapy.utils.datatypes.LocalWeakReferencedCache` (:gh:`4597`,
+ :gh:`4599`)
+
+* When :setting:`FEEDS` defines multiple URIs, log messages about items being
+ stored now contain information from the corresponding feed, instead of
+ always containing information about only one of the feeds (:gh:`4619`,
+ :gh:`4629`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* Added a new section about :ref:`accessing cb_kwargs from errbacks
+ <errback-cb_kwargs>` (:gh:`4598`, :gh:`4634`)
+
+* Covered chompjs_ in :ref:`topics-parsing-javascript` (:gh:`4556`,
+ :gh:`4562`)
+
+* Removed from :doc:`topics/coroutines` the warning about the API being
+ experimental (:gh:`4511`, :gh:`4513`)
+
+* Removed references to unsupported versions of :doc:`Twisted
+ <twisted:index>` (:gh:`4533`)
+
+* Updated the description of the :ref:`screenshot pipeline example
+ <ScreenshotPipeline>`, which now uses :doc:`coroutine syntax
+ <topics/coroutines>` instead of returning a
+ :class:`~twisted.internet.defer.Deferred` (:gh:`4514`, :gh:`4593`)
+
+* Removed a misleading import line from the
+ :func:`scrapy.utils.log.configure_logging` code example (:gh:`4510`,
+ :gh:`4587`)
+
+* The display-on-hover behavior of internal documentation references now also
+ covers links to :ref:`commands <topics-commands>`, :attr:`.Request.meta`
+ keys, :ref:`settings <topics-settings>` and
+ :ref:`signals <topics-signals>` (:gh:`4495`, :gh:`4563`)
+
+* It is again possible to download the documentation for offline reading
+ (:gh:`4578`, :gh:`4585`)
+
+* Removed backslashes preceding ``*args`` and ``**kwargs`` in some function
+ and method signatures (:gh:`4592`, :gh:`4596`)
+
+.. _chompjs: https://github.com/Nykakin/chompjs
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* Adjusted the code base further to our :ref:`style guidelines
+ <coding-style>` (:gh:`4237`, :gh:`4525`, :gh:`4538`,
+ :gh:`4539`, :gh:`4540`, :gh:`4542`, :gh:`4543`, :gh:`4544`,
+ :gh:`4545`, :gh:`4557`, :gh:`4558`, :gh:`4566`, :gh:`4568`,
+ :gh:`4572`)
+
+* Removed remnants of Python 2 support (:gh:`4550`, :gh:`4553`,
+ :gh:`4568`)
+
+* Improved code sharing between the :command:`crawl` and :command:`runspider`
+ commands (:gh:`4548`, :gh:`4552`)
+
+* Replaced ``chain(*iterable)`` with ``chain.from_iterable(iterable)``
+ (:gh:`4635`)
+
+* You may now run the :mod:`asyncio` tests with Tox on any Python version
+ (:gh:`4521`)
+
+* Updated test requirements to reflect an incompatibility with pytest 5.4 and
+ 5.4.1 (:gh:`4588`)
+
+* Improved :class:`~scrapy.spiderloader.SpiderLoader` test coverage for
+ scenarios involving duplicate spider names (:gh:`4549`, :gh:`4560`)
+
+* Configured Travis CI to also run the tests with Python 3.5.2
+ (:gh:`4518`, :gh:`4615`)
+
+* Added a `Pylint <https://www.pylint.org/>`_ job to Travis CI
+ (:gh:`3727`)
+
+* Added a `Mypy <https://mypy-lang.org/>`_ job to Travis CI (:gh:`4637`)
+
+* Made use of set literals in tests (:gh:`4573`)
+
+* Cleaned up the Travis CI configuration (:gh:`4517`, :gh:`4519`,
+ :gh:`4522`, :gh:`4537`)
+
+
+.. _release-2.1.0:
+
+Scrapy 2.1.0 (2020-04-24)
+-------------------------
+
+Highlights:
+
+* New :setting:`FEEDS` setting to export to multiple feeds
+* New :attr:`Response.ip_address <scrapy.http.Response.ip_address>` attribute
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* :exc:`AssertionError` exceptions triggered by :ref:`assert <assert>`
+ statements have been replaced by new exception types, to support running
+ Python in optimized mode (see :option:`-O`) without changing Scrapy’s
+ behavior in any unexpected ways.
+
+ If you catch an :exc:`AssertionError` exception from Scrapy, update your
+ code to catch the corresponding new exception.
+
+ (:gh:`4440`)
+
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* The ``LOG_UNSERIALIZABLE_REQUESTS`` setting is no longer supported, use
+ :setting:`SCHEDULER_DEBUG` instead (:gh:`4385`)
+
+* The ``REDIRECT_MAX_METAREFRESH_DELAY`` setting is no longer supported, use
+ :setting:`METAREFRESH_MAXDELAY` instead (:gh:`4385`)
+
+* The :class:`~scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware`
+ middleware has been removed, including the entire
+ :class:`scrapy.downloadermiddlewares.chunked` module; chunked transfers
+ work out of the box (:gh:`4431`)
+
+* The ``spiders`` property has been removed from
+ :class:`~scrapy.crawler.Crawler`, use :class:`CrawlerRunner.spider_loader
+ <scrapy.crawler.CrawlerRunner.spider_loader>` or instantiate
+ :setting:`SPIDER_LOADER_CLASS` with your settings instead (:gh:`4398`)
+
+* The ``MultiValueDict``, ``MultiValueDictKeyError``, and ``SiteNode``
+ classes have been removed from :mod:`scrapy.utils.datatypes`
+ (:gh:`4400`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* The ``FEED_FORMAT`` and ``FEED_URI`` settings have been deprecated in
+ favor of the new :setting:`FEEDS` setting (:gh:`1336`, :gh:`3858`,
+ :gh:`4507`)
+
+
+New features
+~~~~~~~~~~~~
+
+* A new setting, :setting:`FEEDS`, allows configuring multiple output feeds
+ with different settings each (:gh:`1336`, :gh:`3858`, :gh:`4507`)
+
+* The :command:`crawl` and :command:`runspider` commands now support multiple
+ ``-o`` parameters (:gh:`1336`, :gh:`3858`, :gh:`4507`)
+
+* The :command:`crawl` and :command:`runspider` commands now support
+ specifying an output format by appending ``:<format>`` to the output file
+ (:gh:`1336`, :gh:`3858`, :gh:`4507`)
+
+* The new :attr:`Response.ip_address <scrapy.http.Response.ip_address>`
+ attribute gives access to the IP address that originated a response
+ (:gh:`3903`, :gh:`3940`)
+
+* A warning is now issued when a value in
+ :attr:`~scrapy.spiders.Spider.allowed_domains` includes a port
+ (:gh:`50`, :gh:`3198`, :gh:`4413`)
+
+* Zsh completion now excludes used option aliases from the completion list
+ (:gh:`4438`)
+
+
+Bug fixes
+~~~~~~~~~
+
+* :ref:`Request serialization <request-serialization>` no longer breaks for
+ callbacks that are spider attributes which are assigned a function with a
+ different name (:gh:`4500`)
+
+* ``None`` values in :attr:`~scrapy.spiders.Spider.allowed_domains` no longer
+ cause a :exc:`TypeError` exception (:gh:`4410`)
+
+* Zsh completion no longer allows options after arguments (:gh:`4438`)
+
+* zope.interface 5.0.0 and later versions are now supported
+ (:gh:`4447`, :gh:`4448`)
+
+* ``Spider.make_requests_from_url``, deprecated in Scrapy 1.4.0, now issues a
+ warning when used (:gh:`4412`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* Improved the documentation about signals that allow their handlers to
+ return a :class:`~twisted.internet.defer.Deferred` (:gh:`4295`,
+ :gh:`4390`)
+
+* Our PyPI entry now includes links for our documentation, our source code
+ repository and our issue tracker (:gh:`4456`)
+
+* Covered the `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_
+ service in the documentation (:gh:`4206`, :gh:`4455`)
+
+* Removed references to the Guppy library, which only works in Python 2
+ (:gh:`4285`, :gh:`4343`)
+
+* Extended use of InterSphinx to link to Python 3 documentation
+ (:gh:`4444`, :gh:`4445`)
+
+* Added support for Sphinx 3.0 and later (:gh:`4475`, :gh:`4480`,
+ :gh:`4496`, :gh:`4503`)
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* Removed warnings about using old, removed settings (:gh:`4404`)
+
+* Removed a warning about importing
+ :class:`~twisted.internet.testing.StringTransport` from
+ ``twisted.test.proto_helpers`` in Twisted 19.7.0 or newer (:gh:`4409`)
+
+* Removed outdated Debian package build files (:gh:`4384`)
+
+* Removed :class:`object` usage as a base class (:gh:`4430`)
+
+* Removed code that added support for old versions of Twisted that we no
+ longer support (:gh:`4472`)
+
+* Fixed code style issues (:gh:`4468`, :gh:`4469`, :gh:`4471`,
+ :gh:`4481`)
+
+* Removed :func:`twisted.internet.defer.returnValue` calls (:gh:`4443`,
+ :gh:`4446`, :gh:`4489`)
+
+
+.. _release-2.0.1:
+
+Scrapy 2.0.1 (2020-03-18)
+-------------------------
+
+* :meth:`Response.follow_all <scrapy.http.Response.follow_all>` now supports
+ an empty URL iterable as input (:gh:`4408`, :gh:`4420`)
+
+* Removed top-level :mod:`~twisted.internet.reactor` imports to prevent
+ errors about the wrong Twisted reactor being installed when setting a
+ different Twisted reactor using :setting:`TWISTED_REACTOR` (:gh:`4401`,
+ :gh:`4406`)
+
+* Fixed tests (:gh:`4422`)
+
+
+.. _release-2.0.0:
+
+Scrapy 2.0.0 (2020-03-03)
+-------------------------
+
+Highlights:
+
+* Python 2 support has been removed
+* :doc:`Partial <topics/coroutines>` :ref:`coroutine syntax <async>` support
+ and :doc:`experimental <topics/asyncio>` :mod:`asyncio` support
+* New :meth:`Response.follow_all <scrapy.http.Response.follow_all>` method
+* :ref:`FTP support <media-pipeline-ftp>` for media pipelines
+* New :attr:`Response.certificate <scrapy.http.Response.certificate>`
+ attribute
+* IPv6 support through ``DNS_RESOLVER``
+
+Backward-incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* Python 2 support has been removed, following `Python 2 end-of-life on
+ January 1, 2020`_ (:gh:`4091`, :gh:`4114`, :gh:`4115`,
+ :gh:`4121`, :gh:`4138`, :gh:`4231`, :gh:`4242`, :gh:`4304`,
+ :gh:`4309`, :gh:`4373`)
+
+* Retry gaveups (see :setting:`RETRY_TIMES`) are now logged as errors instead
+ of as debug information (:gh:`3171`, :gh:`3566`)
+
+* File extensions that
+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ ignores by default now also include ``7z``, ``7zip``, ``apk``, ``bz2``,
+ ``cdr``, ``dmg``, ``ico``, ``iso``, ``tar``, ``tar.gz``, ``webm``, and
+ ``xz`` (:gh:`1837`, :gh:`2067`, :gh:`4066`)
+
+* The :setting:`METAREFRESH_IGNORE_TAGS` setting is now an empty list by
+ default, following web browser behavior (:gh:`3844`, :gh:`4311`)
+
+* The
+ :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
+ now includes spaces after commas in the value of the ``Accept-Encoding``
+ header that it sets, following web browser behavior (:gh:`4293`)
+
+* The ``__init__`` method of custom download handlers (see
+ :setting:`DOWNLOAD_HANDLERS`) or subclasses of the following downloader
+ handlers no longer receives a ``settings`` parameter:
+
+ * :class:`scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler`
+
+ * :class:`scrapy.core.downloader.handlers.file.FileDownloadHandler`
+
+ Use the ``from_settings`` or ``from_crawler`` class methods to expose such
+ a parameter to your custom download handlers.
+
+ (:gh:`4126`)
+
+* We have refactored the :class:`scrapy.core.scheduler.Scheduler` class and
+ related queue classes (see :setting:`SCHEDULER_PRIORITY_QUEUE`,
+ :setting:`SCHEDULER_DISK_QUEUE` and :setting:`SCHEDULER_MEMORY_QUEUE`) to
+ make it easier to implement custom scheduler queue classes. See
+ :ref:`2-0-0-scheduler-queue-changes` below for details.
+
+* Overridden settings are now logged in a different format. This is more in
+ line with similar information logged at startup (:gh:`4199`)
+
+.. _Python 2 end-of-life on January 1, 2020: https://www.python.org/doc/sunset-python-2/
+
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* The :ref:`Scrapy shell <topics-shell>` no longer provides a `sel` proxy
+ object, use :meth:`response.selector <scrapy.http.TextResponse.selector>`
+ instead (:gh:`4347`)
+
+* LevelDB support has been removed (:gh:`4112`)
+
+* The following functions have been removed from :mod:`scrapy.utils.python`:
+ ``isbinarytext``, ``is_writable``, ``setattr_default``, ``stringify_dict``
+ (:gh:`4362`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* Using environment variables prefixed with ``SCRAPY_`` to override settings
+ is deprecated (:gh:`4300`, :gh:`4374`, :gh:`4375`)
+
+* :class:`scrapy.linkextractors.FilteringLinkExtractor` is deprecated, use
+ :class:`scrapy.linkextractors.LinkExtractor
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` instead (:gh:`4045`)
+
+* The ``noconnect`` query string argument of proxy URLs is deprecated and
+ should be removed from proxy URLs (:gh:`4198`)
+
+* The :meth:`next <scrapy.utils.python.MutableChain.next>` method of
+ :class:`scrapy.utils.python.MutableChain` is deprecated, use the global
+ :func:`next` function or :meth:`MutableChain.__next__
+ <scrapy.utils.python.MutableChain.__next__>` instead (:gh:`4153`)
+
+
+New features
+~~~~~~~~~~~~
+
+* Added :doc:`partial support <topics/coroutines>` for Python’s
+ :ref:`coroutine syntax <async>` and :doc:`experimental support
+ <topics/asyncio>` for :mod:`asyncio` and :mod:`asyncio`-powered libraries
+ (:gh:`4010`, :gh:`4259`, :gh:`4269`, :gh:`4270`, :gh:`4271`,
+ :gh:`4316`, :gh:`4318`)
+
+* The new :meth:`Response.follow_all <scrapy.http.Response.follow_all>`
+ method offers the same functionality as
+ :meth:`Response.follow <scrapy.http.Response.follow>` but supports an
+ iterable of URLs as input and returns an iterable of requests
+ (:gh:`2582`, :gh:`4057`, :gh:`4286`)
+
+* :ref:`Media pipelines <topics-media-pipeline>` now support :ref:`FTP
+ storage <media-pipeline-ftp>` (:gh:`3928`, :gh:`3961`)
+
+* The new :attr:`Response.certificate <scrapy.http.Response.certificate>`
+ attribute exposes the SSL certificate of the server as a
+ :class:`twisted.internet.ssl.Certificate` object for HTTPS responses
+ (:gh:`2726`, :gh:`4054`)
+
+* A new ``DNS_RESOLVER`` setting allows enabling IPv6 support
+ (:gh:`1031`, :gh:`4227`)
+
+* A new :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` setting allows configuring
+ the existing soft limit that pauses request downloads when the total
+ response data being processed is too high (:gh:`1410`, :gh:`3551`)
+
+* A new :setting:`TWISTED_REACTOR` setting allows customizing the
+ :mod:`~twisted.internet.reactor` that Scrapy uses, allowing to
+ :doc:`enable asyncio support <topics/asyncio>` or deal with a
+ :ref:`common macOS issue <faq-specific-reactor>` (:gh:`2905`,
+ :gh:`4294`)
+
+* Scheduler disk and memory queues may now use the class methods
+ ``from_crawler`` or ``from_settings`` (:gh:`3884`)
+
+* The new :attr:`Response.cb_kwargs <scrapy.http.Response.cb_kwargs>`
+ attribute serves as a shortcut for :attr:`Response.request.cb_kwargs
+ <scrapy.Request.cb_kwargs>` (:gh:`4331`)
+
+* :meth:`Response.follow <scrapy.http.Response.follow>` now supports a
+ ``flags`` parameter, for consistency with :class:`~scrapy.Request`
+ (:gh:`4277`, :gh:`4279`)
+
+* :ref:`Item loader processors <topics-loaders-processors>` can now be
+ regular functions, they no longer need to be methods (:gh:`3899`)
+
+* :class:`~scrapy.spiders.Rule` now accepts an ``errback`` parameter
+ (:gh:`4000`)
+
+* :class:`~scrapy.Request` no longer requires a ``callback`` parameter
+ when an ``errback`` parameter is specified (:gh:`3586`, :gh:`4008`)
+
+* :class:`~scrapy.logformatter.LogFormatter` now supports some additional
+ methods:
+
+ * :class:`~scrapy.logformatter.LogFormatter.download_error` for
+ download errors
+
+ * :class:`~scrapy.logformatter.LogFormatter.item_error` for exceptions
+ raised during item processing by :ref:`item pipelines
+ <topics-item-pipeline>`
+
+ * :class:`~scrapy.logformatter.LogFormatter.spider_error` for exceptions
+ raised from :ref:`spider callbacks <topics-spiders>`
+
+ (:gh:`374`, :gh:`3986`, :gh:`3989`, :gh:`4176`, :gh:`4188`)
+
+* The :setting:`FEED_URI` setting now supports :class:`pathlib.Path` values
+ (:gh:`3731`, :gh:`4074`)
+
+* A new :signal:`request_left_downloader` signal is sent when a request
+ leaves the downloader (:gh:`4303`)
+
+* Scrapy logs a warning when it detects a request callback or errback that
+ uses ``yield`` but also returns a value, since the returned value would be
+ lost (:gh:`3484`, :gh:`3869`)
+
+* :class:`~scrapy.spiders.Spider` objects now raise an :exc:`AttributeError`
+ exception if they do not have a :class:`~scrapy.spiders.Spider.start_urls`
+ attribute nor reimplement ``scrapy.spiders.Spider.start_requests()``,
+ but have a ``start_url`` attribute (:gh:`4133`, :gh:`4170`)
+
+* :class:`~scrapy.exporters.BaseItemExporter` subclasses may now use
+ ``super().__init__(**kwargs)`` instead of ``self._configure(kwargs)`` in
+ their ``__init__`` method, passing ``dont_fail=True`` to the parent
+ ``__init__`` method if needed, and accessing ``kwargs`` at ``self._kwargs``
+ after calling their parent ``__init__`` method (:gh:`4193`,
+ :gh:`4370`)
+
+* A new ``keep_fragments`` parameter of
+ ``scrapy.utils.request.request_fingerprint`` allows to generate
+ different fingerprints for requests with different fragments in their URL
+ (:gh:`4104`)
+
+* Download handlers (see :setting:`DOWNLOAD_HANDLERS`) may now use the
+ ``from_settings`` and ``from_crawler`` class methods that other Scrapy
+ components already supported (:gh:`4126`)
+
+* :class:`scrapy.utils.python.MutableChain.__iter__` now returns ``self``,
+ allowing it to be used as a sequence.
+ (:gh:`4153`)
+
+
+Bug fixes
+~~~~~~~~~
+
+* The :command:`crawl` command now also exits with exit code 1 when an
+ exception happens before the crawling starts (:gh:`4175`, :gh:`4207`)
+
+* :class:`LinkExtractor.extract_links
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` no longer
+ re-encodes the query string or URLs from non-UTF-8 responses in UTF-8
+ (:gh:`998`, :gh:`1403`, :gh:`1949`, :gh:`4321`)
+
+* The first spider middleware (see :setting:`SPIDER_MIDDLEWARES`) now also
+ processes exceptions raised from callbacks that are generators
+ (:gh:`4260`, :gh:`4272`)
+
+* Redirects to URLs starting with 3 slashes (``///``) are now supported
+ (:gh:`4032`, :gh:`4042`)
+
+* :class:`~scrapy.Request` no longer accepts strings as ``url`` simply
+ because they have a colon (:gh:`2552`, :gh:`4094`)
+
+* The correct encoding is now used for attach names in
+ :class:`~scrapy.mail.MailSender` (:gh:`4229`, :gh:`4239`)
+
+* :class:`~scrapy.dupefilters.RFPDupeFilter`, the default
+ :setting:`DUPEFILTER_CLASS`, no longer writes an extra ``\r`` character on
+ each line in Windows, which made the size of the ``requests.seen`` file
+ unnecessarily large on that platform (:gh:`4283`)
+
+* Z shell auto-completion now looks for ``.html`` files, not ``.http`` files,
+ and covers the ``-h`` command-line switch (:gh:`4122`, :gh:`4291`)
+
+* Adding items to a :class:`scrapy.utils.datatypes.LocalCache` object
+ without a ``limit`` defined no longer raises a :exc:`TypeError` exception
+ (:gh:`4123`)
+
+* Fixed a typo in the message of the :exc:`ValueError` exception raised when
+ :func:`scrapy.utils.misc.create_instance` gets both ``settings`` and
+ ``crawler`` set to ``None`` (:gh:`4128`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* API documentation now links to an online, syntax-highlighted view of the
+ corresponding source code (:gh:`4148`)
+
+* Links to unexisting documentation pages now allow access to the sidebar
+ (:gh:`4152`, :gh:`4169`)
+
+* Cross-references within our documentation now display a tooltip when
+ hovered (:gh:`4173`, :gh:`4183`)
+
+* Improved the documentation about :meth:`LinkExtractor.extract_links
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` and
+ simplified :ref:`topics-link-extractors` (:gh:`4045`)
+
+* Clarified how :class:`ItemLoader.item <scrapy.loader.ItemLoader.item>`
+ works (:gh:`3574`, :gh:`4099`)
+
+* Clarified that :func:`logging.basicConfig` should not be used when also
+ using :class:`~scrapy.crawler.CrawlerProcess` (:gh:`2149`,
+ :gh:`2352`, :gh:`3146`, :gh:`3960`)
+
+* Clarified the requirements for :class:`~scrapy.Request` objects
+ :ref:`when using persistence <request-serialization>` (:gh:`4124`,
+ :gh:`4139`)
+
+* Clarified how to install a :ref:`custom image pipeline
+ <media-pipeline-example>` (:gh:`4034`, :gh:`4252`)
+
+* Fixed the signatures of the ``file_path`` method in :ref:`media pipeline
+ <topics-media-pipeline>` examples (:gh:`4290`)
+
+* Covered a backward-incompatible change in Scrapy 1.7.0 affecting custom
+ :class:`scrapy.core.scheduler.Scheduler` subclasses (:gh:`4274`)
+
+* Improved the ``README.rst`` and ``CODE_OF_CONDUCT.md`` files
+ (:gh:`4059`)
+
+* Documentation examples are now checked as part of our test suite and we
+ have fixed some of the issues detected (:gh:`4142`, :gh:`4146`,
+ :gh:`4171`, :gh:`4184`, :gh:`4190`)
+
+* Fixed logic issues, broken links and typos (:gh:`4247`, :gh:`4258`,
+ :gh:`4282`, :gh:`4288`, :gh:`4305`, :gh:`4308`, :gh:`4323`,
+ :gh:`4338`, :gh:`4359`, :gh:`4361`)
+
+* Improved consistency when referring to the ``__init__`` method of an object
+ (:gh:`4086`, :gh:`4088`)
+
+* Fixed an inconsistency between code and output in :ref:`intro-overview`
+ (:gh:`4213`)
+
+* Extended :mod:`~sphinx.ext.intersphinx` usage (:gh:`4147`,
+ :gh:`4172`, :gh:`4185`, :gh:`4194`, :gh:`4197`)
+
+* We now use a recent version of Python to build the documentation
+ (:gh:`4140`, :gh:`4249`)
+
+* Cleaned up documentation (:gh:`4143`, :gh:`4275`)
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* Re-enabled proxy ``CONNECT`` tests (:gh:`2545`, :gh:`4114`)
+
+* Added Bandit_ security checks to our test suite (:gh:`4162`,
+ :gh:`4181`)
+
+* Added Flake8_ style checks to our test suite and applied many of the
+ corresponding changes (:gh:`3944`, :gh:`3945`, :gh:`4137`,
+ :gh:`4157`, :gh:`4167`, :gh:`4174`, :gh:`4186`, :gh:`4195`,
+ :gh:`4238`, :gh:`4246`, :gh:`4355`, :gh:`4360`, :gh:`4365`)
+
+* Improved test coverage (:gh:`4097`, :gh:`4218`, :gh:`4236`)
+
+* Started reporting slowest tests, and improved the performance of some of
+ them (:gh:`4163`, :gh:`4164`)
+
+* Fixed broken tests and refactored some tests (:gh:`4014`, :gh:`4095`,
+ :gh:`4244`, :gh:`4268`, :gh:`4372`)
+
+* Modified the :doc:`tox <tox:index>` configuration to allow running tests
+ with any Python version, run Bandit_ and Flake8_ tests by default, and
+ enforce a minimum tox version programmatically (:gh:`4179`)
+
+* Cleaned up code (:gh:`3937`, :gh:`4208`, :gh:`4209`,
+ :gh:`4210`, :gh:`4212`, :gh:`4369`, :gh:`4376`, :gh:`4378`)
+
+.. _Bandit: https://bandit.readthedocs.io/en/latest/
+.. _Flake8: https://flake8.pycqa.org/en/latest/
+
+
+.. _2-0-0-scheduler-queue-changes:
+
+Changes to scheduler queue classes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following changes may impact any custom queue classes of all types:
+
+* The ``push`` method no longer receives a second positional parameter
+ containing ``request.priority * -1``. If you need that value, get it
+ from the first positional parameter, ``request``, instead, or use
+ the new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
+ method in :class:`scrapy.core.scheduler.ScrapyPriorityQueue`
+ subclasses.
+
+The following changes may impact custom priority queue classes:
+
+* In the ``__init__`` method or the ``from_crawler`` or ``from_settings``
+ class methods:
+
+ * The parameter that used to contain a factory function,
+ ``qfactory``, is now passed as a keyword parameter named
+ ``downstream_queue_cls``.
+
+ * A new keyword parameter has been added: ``key``. It is a string
+ that is always an empty string for memory queues and indicates the
+ :setting:`JOBDIR` value for disk queues.
+
+ * The parameter for disk queues that contains data from the previous
+ crawl, ``startprios`` or ``slot_startprios``, is now passed as a
+ keyword parameter named ``startprios``.
+
+ * The ``serialize`` parameter is no longer passed. The disk queue
+ class must take care of request serialization on its own before
+ writing to disk, using the
+ :func:`~scrapy.utils.reqser.request_to_dict` and
+ :func:`~scrapy.utils.reqser.request_from_dict` functions from the
+ :mod:`scrapy.utils.reqser` module.
+
+The following changes may impact custom disk and memory queue classes:
+
+* The signature of the ``__init__`` method is now
+ ``__init__(self, crawler, key)``.
+
+The following changes affect specifically the
+:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` and
+:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` classes from
+:mod:`scrapy.core.scheduler` and may affect subclasses:
+
+* In the ``__init__`` method, most of the changes described above apply.
+
+ ``__init__`` may still receive all parameters as positional parameters,
+ however:
+
+ * ``downstream_queue_cls``, which replaced ``qfactory``, must be
+ instantiated differently.
+
+ ``qfactory`` was instantiated with a priority value (integer).
+
+ Instances of ``downstream_queue_cls`` should be created using
+ the new
+ :meth:`ScrapyPriorityQueue.qfactory <scrapy.core.scheduler.ScrapyPriorityQueue.qfactory>`
+ or
+ :meth:`DownloaderAwarePriorityQueue.pqfactory <scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqfactory>`
+ methods.
+
+ * The new ``key`` parameter displaced the ``startprios``
+ parameter 1 position to the right.
+
+* The following class attributes have been added:
+
+ * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.crawler`
+
+ * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.downstream_queue_cls`
+ (details above)
+
+ * :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.key` (details above)
+
+* The ``serialize`` attribute has been removed (details above)
+
+The following changes affect specifically the
+:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` class and may affect
+subclasses:
+
+* A new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
+ method has been added which, given a request, returns
+ ``request.priority * -1``.
+
+ It is used in :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.push`
+ to make up for the removal of its ``priority`` parameter.
+
+* The ``spider`` attribute has been removed. Use
+ :attr:`crawler.spider <scrapy.core.scheduler.ScrapyPriorityQueue.crawler>`
+ instead.
+
+The following changes affect specifically the
+:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` class and may
+affect subclasses:
+
+* A new :attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqueues`
+ attribute offers a mapping of downloader slot names to the
+ corresponding instances of
+ :attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.downstream_queue_cls`.
+
+(:gh:`3884`)
+
+
+.. _boto3: https://github.com/boto/boto3
+.. _botocore: https://github.com/boto/botocore
+.. _brotlipy: https://github.com/python-hyper/brotlipy/
+.. _cryptography: https://cryptography.io/en/latest/
+.. _lxml: https://lxml.de/
+.. _parsel: https://github.com/scrapy/parsel
+.. _Pillow: https://github.com/python-pillow/Pillow
+.. _pyOpenSSL: https://www.pyopenssl.org/en/stable/
+.. _scrapy-bench: https://github.com/scrapy/scrapy-bench
+.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash
+.. _service_identity: https://service-identity.readthedocs.io/en/stable/
+.. _Twisted: https://twisted.org/
+.. _zope.interface: https://zopeinterface.readthedocs.io/en/latest/
+.. _zstandard: https://pypi.org/project/zstandard/