Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708)

Adrián Chaves3600582f5607
diff --git a/docs/news.rst b/docs/news.rst
index 80d130e4a..850b323ef 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -3,6 +3,139 @@
Release notes
=============
+.. _release-2.3.0:
+
+Scrapy 2.3.0 (2020-08-04)
+-------------------------
+
+Highlights:
+
+* :ref:`Feed exports <topics-feed-exports>` now support :ref:`Google Cloud
+ Storage <topics-feed-storage-gcs>` as a storage backend
+
+* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver
+ output items in batches of up to the specified number of items.
+
+ It also serves as a workaround for :ref:`delayed file delivery
+ <delayed-file-delivery>`, which causes Scrapy to only start item delivery
+ after the crawl has finished when using certain storage backends
+ (:ref:`S3 <topics-feed-storage-s3>`, :ref:`FTP <topics-feed-storage-ftp>`,
+ and now :ref:`GCS <topics-feed-storage-gcs>`).
+
+* The base implementation of :ref:`item loaders <topics-loaders>` has been
+ moved into a separate library, :doc:`itemloaders <itemloaders:index>`,
+ allowing usage from outside Scrapy and a separate release schedule
+
+Deprecation removals
+~~~~~~~~~~~~~~~~~~~~
+
+* Removed the following classes and their parent modules from
+ ``scrapy.linkextractors``:
+
+ * ``htmlparser.HtmlParserLinkExtractor``
+ * ``regex.RegexLinkExtractor``
+ * ``sgml.BaseSgmlLinkExtractor``
+ * ``sgml.SgmlLinkExtractor``
+
+ Use
+ :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
+ instead (:issue:`4356`, :issue:`4679`)
+
+
+Deprecations
+~~~~~~~~~~~~
+
+* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated
+ (:issue:`4683`)
+
+
+New features
+~~~~~~~~~~~~
+
+* :ref:`Feed exports <topics-feed-exports>` support :ref:`Google Cloud
+ Storage <topics-feed-storage-gcs>` (:issue:`685`, :issue:`3608`)
+
+* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries
+ (:issue:`4250`, :issue:`4434`)
+
+* The :command:`parse` command now allows specifying an output file
+ (:issue:`4317`, :issue:`4377`)
+
+* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
+ ``--data-raw`` (:issue:`4612`)
+
+* A ``parse`` callback may now be used in built-in spider subclasses, such
+ as :class:`~scrapy.spiders.CrawlSpider` (:issue:`712`, :issue:`732`,
+ :issue:`781`, :issue:`4254` )
+
+
+Bug fixes
+~~~~~~~~~
+
+* Fixed the :ref:`CSV exporting <topics-feed-format-csv>` of
+ :ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
+ <attrs-items>` (:issue:`4667`, :issue:`4668`)
+
+* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
+ :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
+ method to ``POST`` when a request body is specified and no request method
+ is specified (:issue:`4612`)
+
+* The processing of ANSI escape sequences in enabled in Windows 10.0.14393
+ and later, where it is required for colored output (:issue:`4393`,
+ :issue:`4403`)
+
+
+Documentation
+~~~~~~~~~~~~~
+
+* Updated the `OpenSSL cipher list format`_ link in the documentation about
+ the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:issue:`4653`)
+
+* Simplified the code example in :ref:`topics-loaders-dataclass`
+ (:issue:`4652`)
+
+.. _OpenSSL cipher list format: https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html#CIPHER-LIST-FORMAT
+
+
+Quality assurance
+~~~~~~~~~~~~~~~~~
+
+* The base implementation of :ref:`item loaders <topics-loaders>` has been
+ moved into :doc:`itemloaders <itemloaders:index>` (:issue:`4005`,
+ :issue:`4516`)
+
+* Fixed a silenced error in some scheduler tests (:issue:`4644`,
+ :issue:`4645`)
+
+* Renewed the localhost certificate used for SSL tests (:issue:`4650`)
+
+* Removed cookie-handling code specific to Python 2 (:issue:`4682`)
+
+* Stopped using Python 2 unicode literal syntax (:issue:`4704`)
+
+* Stopped using a backlash for line continuation (:issue:`4673`)
+
+* Removed unneeded entries from the MyPy exception list (:issue:`4690`)
+
+* Automated tests now pass on Windows as part of our continuous integration
+ system (:issue:`4458`)
+
+* Automated tests now pass on the latest PyPy version for supported Python
+ versions in our continuous integration system (:issue:`4504`)
+
+
+.. _release-2.2.1:
+
+Scrapy 2.2.1 (2020-07-17)
+-------------------------
+
+* The :command:`startproject` command no longer makes unintended changes to
+ the permissions of files in the destination folder, such as removing
+ execution permissions (:issue:`4662`, :issue:`4666`)
+
+
.. _release-2.2.0:
Scrapy 2.2.0 (2020-06-24)
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index 4fce51abc..9638a2322 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -493,6 +493,8 @@ Supported options:
* ``--output`` or ``-o``: dump scraped items to a file
+ .. versionadded:: 2.3
+
.. skip: start
Usage example::
diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst
index 4e87a00f2..101aa159c 100644
--- a/docs/topics/developer-tools.rst
+++ b/docs/topics/developer-tools.rst
@@ -289,8 +289,10 @@ request::
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")
Alternatively, if you want to know the arguments needed to recreate that
-request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
-function to get a dictionary with the equivalent arguments.
+request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`
+function to get a dictionary with the equivalent arguments:
+
+.. autofunction:: scrapy.utils.curl.curl_to_request_kwargs
Note that to translate a cURL command into a Scrapy request,
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index dd4eb3c61..37b7096f6 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -100,6 +100,7 @@ The storages backends supported out of the box are:
* :ref:`topics-feed-storage-fs`
* :ref:`topics-feed-storage-ftp`
* :ref:`topics-feed-storage-s3` (requires botocore_)
+ * :ref:`topics-feed-storage-gcs` (requires `google-cloud-storage`_)
* :ref:`topics-feed-storage-stdout`
Some storage backends may be unavailable if the required external libraries are
@@ -169,6 +170,9 @@ FTP supports two different connection modes: `active or passive
mode by default. To use the active connection mode instead, set the
:setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.
+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
+
+
.. _topics-feed-storage-s3:
S3
@@ -194,11 +198,16 @@ You can also define a custom ACL for exported feeds using this setting:
* :setting:`FEED_STORAGE_S3_ACL`
+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
+
+
.. _topics-feed-storage-gcs:
Google Cloud Storage (GCS)
--------------------------
+.. versionadded:: 2.3
+
The feeds are stored on `Google Cloud Storage`_.
* URI scheme: ``gs``
@@ -206,7 +215,7 @@ The feeds are stored on `Google Cloud Storage`_.
* ``gs://mybucket/path/to/export.csv``
- * Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.
+ * Required external libraries: `google-cloud-storage`_.
For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.
@@ -215,6 +224,11 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following
* :setting:`FEED_STORAGE_GCS_ACL`
* :setting:`GCS_PROJECT_ID`
+This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
+
+.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
+
+
.. _topics-feed-storage-stdout:
Standard output
@@ -227,6 +241,26 @@ The feeds are written to the standard output of the Scrapy process.
* Required external libraries: none
+.. _delayed-file-delivery:
+
+Delayed file delivery
+---------------------
+
+As indicated above, some of the described storage backends use delayed file
+delivery.
+
+These storage backends do not upload items to the feed URI as those items are
+scraped. Instead, Scrapy writes items into a temporary local file, and only
+once all the file contents have been written (i.e. at the end of the crawl) is
+that file uploaded to the feed URI.
+
+If you want item delivery to start earlier when using one of these storage
+backends, use :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items
+in multiple files, with the specified maximum item count per file. That way, as
+soon as a file reaches the maximum item count, that file is delivered to the
+feed URI, allowing item delivery to start way before the end of the crawl.
+
+
Settings
========
diff --git a/scrapy/utils/curl.py b/scrapy/utils/curl.py
index aa681522f..9c0efcec4 100644
--- a/scrapy/utils/curl.py
+++ b/scrapy/utils/curl.py
@@ -39,7 +39,8 @@ def curl_to_request_kwargs(curl_command, ignore_unknown_options=True):
:param str curl_command: string containing the curl command
:param bool ignore_unknown_options: If true, only a warning is emitted when
- cURL options are unknown. Otherwise raises an error. (default: True)
+ cURL options are unknown. Otherwise
+ raises an error. (default: True)
:return: dictionary of Request kwargs
"""