Scrapy, a fast high-level web crawling & scraping framework for Python.
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708)
Adrián Chaves3600582f5607
diff --git a/docs/news.rst b/docs/news.rstindex 80d130e4a..850b323ef 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,6 +3,139 @@Release notes=============+.. _release-2.3.0:++Scrapy 2.3.0 (2020-08-04)+-------------------------++Highlights:++* :ref:`Feed exports <topics-feed-exports>` now support :ref:`Google Cloud+ Storage <topics-feed-storage-gcs>` as a storage backend++* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver+ output items in batches of up to the specified number of items.++ It also serves as a workaround for :ref:`delayed file delivery+ <delayed-file-delivery>`, which causes Scrapy to only start item delivery+ after the crawl has finished when using certain storage backends+ (:ref:`S3 <topics-feed-storage-s3>`, :ref:`FTP <topics-feed-storage-ftp>`,+ and now :ref:`GCS <topics-feed-storage-gcs>`).++* The base implementation of :ref:`item loaders <topics-loaders>` has been+ moved into a separate library, :doc:`itemloaders <itemloaders:index>`,+ allowing usage from outside Scrapy and a separate release schedule++Deprecation removals+~~~~~~~~~~~~~~~~~~~~++* Removed the following classes and their parent modules from+ ``scrapy.linkextractors``:++ * ``htmlparser.HtmlParserLinkExtractor``+ * ``regex.RegexLinkExtractor``+ * ``sgml.BaseSgmlLinkExtractor``+ * ``sgml.SgmlLinkExtractor``++ Use+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`+ instead (:issue:`4356`, :issue:`4679`)+++Deprecations+~~~~~~~~~~~~++* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated+ (:issue:`4683`)+++New features+~~~~~~~~~~~~++* :ref:`Feed exports <topics-feed-exports>` support :ref:`Google Cloud+ Storage <topics-feed-storage-gcs>` (:issue:`685`, :issue:`3608`)++* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries+ (:issue:`4250`, :issue:`4434`)++* The :command:`parse` command now allows specifying an output file+ (:issue:`4317`, :issue:`4377`)++* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support+ ``--data-raw`` (:issue:`4612`)++* A ``parse`` callback may now be used in built-in spider subclasses, such+ as :class:`~scrapy.spiders.CrawlSpider` (:issue:`712`, :issue:`732`,+ :issue:`781`, :issue:`4254` )+++Bug fixes+~~~~~~~~~++* Fixed the :ref:`CSV exporting <topics-feed-format-csv>` of+ :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items+ <attrs-items>` (:issue:`4667`, :issue:`4668`)++* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request+ method to ``POST`` when a request body is specified and no request method+ is specified (:issue:`4612`)++* The processing of ANSI escape sequences in enabled in Windows 10.0.14393+ and later, where it is required for colored output (:issue:`4393`,+ :issue:`4403`)+++Documentation+~~~~~~~~~~~~~++* Updated the `OpenSSL cipher list format`_ link in the documentation about+ the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:issue:`4653`)++* Simplified the code example in :ref:`topics-loaders-dataclass`+ (:issue:`4652`)++.. _OpenSSL cipher list format: https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html#CIPHER-LIST-FORMAT+++Quality assurance+~~~~~~~~~~~~~~~~~++* The base implementation of :ref:`item loaders <topics-loaders>` has been+ moved into :doc:`itemloaders <itemloaders:index>` (:issue:`4005`,+ :issue:`4516`)++* Fixed a silenced error in some scheduler tests (:issue:`4644`,+ :issue:`4645`)++* Renewed the localhost certificate used for SSL tests (:issue:`4650`)++* Removed cookie-handling code specific to Python 2 (:issue:`4682`)++* Stopped using Python 2 unicode literal syntax (:issue:`4704`)++* Stopped using a backlash for line continuation (:issue:`4673`)++* Removed unneeded entries from the MyPy exception list (:issue:`4690`)++* Automated tests now pass on Windows as part of our continuous integration+ system (:issue:`4458`)++* Automated tests now pass on the latest PyPy version for supported Python+ versions in our continuous integration system (:issue:`4504`)+++.. _release-2.2.1:++Scrapy 2.2.1 (2020-07-17)+-------------------------++* The :command:`startproject` command no longer makes unintended changes to+ the permissions of files in the destination folder, such as removing+ execution permissions (:issue:`4662`, :issue:`4666`)++.. _release-2.2.0:Scrapy 2.2.0 (2020-06-24)diff --git a/docs/topics/commands.rst b/docs/topics/commands.rstindex 4fce51abc..9638a2322 100644--- a/docs/topics/commands.rst+++ b/docs/topics/commands.rst@@ -493,6 +493,8 @@ Supported options:* ``--output`` or ``-o``: dump scraped items to a file+ .. versionadded:: 2.3+.. skip: startUsage example::diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex 4e87a00f2..101aa159c 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -289,8 +289,10 @@ request::"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")Alternatively, if you want to know the arguments needed to recreate that-request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`-function to get a dictionary with the equivalent arguments.+request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`+function to get a dictionary with the equivalent arguments:++.. autofunction:: scrapy.utils.curl.curl_to_request_kwargsNote that to translate a cURL command into a Scrapy request,you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex dd4eb3c61..37b7096f6 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -100,6 +100,7 @@ The storages backends supported out of the box are:* :ref:`topics-feed-storage-fs`* :ref:`topics-feed-storage-ftp`* :ref:`topics-feed-storage-s3` (requires botocore_)+ * :ref:`topics-feed-storage-gcs` (requires `google-cloud-storage`_)* :ref:`topics-feed-storage-stdout`Some storage backends may be unavailable if the required external libraries are@@ -169,6 +170,9 @@ FTP supports two different connection modes: `active or passivemode by default. To use the active connection mode instead, set the:setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.++.. _topics-feed-storage-s3:S3@@ -194,11 +198,16 @@ You can also define a custom ACL for exported feeds using this setting:* :setting:`FEED_STORAGE_S3_ACL`+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.++.. _topics-feed-storage-gcs:Google Cloud Storage (GCS)--------------------------+.. versionadded:: 2.3+The feeds are stored on `Google Cloud Storage`_.* URI scheme: ``gs``@@ -206,7 +215,7 @@ The feeds are stored on `Google Cloud Storage`_.* ``gs://mybucket/path/to/export.csv``- * Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.+ * Required external libraries: `google-cloud-storage`_.For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.@@ -215,6 +224,11 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following* :setting:`FEED_STORAGE_GCS_ACL`* :setting:`GCS_PROJECT_ID`+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.++.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python++.. _topics-feed-storage-stdout:Standard output@@ -227,6 +241,26 @@ The feeds are written to the standard output of the Scrapy process.* Required external libraries: none+.. _delayed-file-delivery:++Delayed file delivery+---------------------++As indicated above, some of the described storage backends use delayed file+delivery.++These storage backends do not upload items to the feed URI as those items are+scraped. Instead, Scrapy writes items into a temporary local file, and only+once all the file contents have been written (i.e. at the end of the crawl) is+that file uploaded to the feed URI.++If you want item delivery to start earlier when using one of these storage+backends, use :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items+in multiple files, with the specified maximum item count per file. That way, as+soon as a file reaches the maximum item count, that file is delivered to the+feed URI, allowing item delivery to start way before the end of the crawl.++Settings========diff --git a/scrapy/utils/curl.py b/scrapy/utils/curl.pyindex aa681522f..9c0efcec4 100644--- a/scrapy/utils/curl.py+++ b/scrapy/utils/curl.py@@ -39,7 +39,8 @@ def curl_to_request_kwargs(curl_command, ignore_unknown_options=True)::param str curl_command: string containing the curl command:param bool ignore_unknown_options: If true, only a warning is emitted when- cURL options are unknown. Otherwise raises an error. (default: True)+ cURL options are unknown. Otherwise+ raises an error. (default: True):return: dictionary of Request kwargs"""