Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Adding documentation about Google Cloud Storage Feed Export

Júlio César Batistacb5f800b0f70
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index b64dbfbfd..efb63b0ba 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -185,6 +185,27 @@ passed through the following settings:
* :setting:`AWS_ACCESS_KEY_ID`
* :setting:`AWS_SECRET_ACCESS_KEY`
+.. _topics-feed-storage-gcs:
+
+Google Cloud Storage (GCS)
+--------------------------
+
+The feeds are stored on `Google Cloud Storage`_.
+
+ * URI scheme: ``gcs``
+ * Example URIs:
+
+ * ``gcs://mybucket/path/to/export.csv``
+
+ * Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.
+
+For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.
+
+You can set a *Project ID* and *Access Control List (ACL)* through the following settings:
+
+ * :setting:`FEED_STORAGE_GCS_ACL`
+ * :setting:`GCS_PROJECT_ID`
+
.. _topics-feed-storage-stdout:
Standard output
@@ -366,3 +387,4 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
.. _Amazon S3: https://aws.amazon.com/s3/
.. _boto: https://github.com/boto/boto
.. _botocore: https://github.com/boto/botocore
+.. _Google Cloud Storage: https://cloud.google.com/storage/
\ No newline at end of file
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 0ac26a9bd..90ae8fd93 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -749,6 +749,14 @@ The Feed Temp dir allows you to set a custom folder to save crawler
temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
:ref:`Amazon S3 <topics-feed-storage-s3>`.
+.. setting:: FEED_STORAGE_GCS_ACL
+
+FEED_STORAGE_GCS_ACL
+--------------------
+
+The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
+For more information on how to set this value, please refer to `Google Cloud documentation <https://cloud.google.com/storage/docs/access-control/lists>`_.
+
.. setting:: FTP_PASSIVE_MODE
FTP_PASSIVE_MODE
@@ -786,6 +794,15 @@ Default: ``"anonymous"``
The username to use for FTP connections when there is no ``"ftp_user"``
in ``Request`` meta.
+.. setting:: GCS_PROJECT_ID
+
+GCS_PROJECT_ID
+-----------------
+
+Default: ``None``
+
+The Project ID that will be used when storing data on `Google Cloud Storage`_.
+
.. setting:: ITEM_PIPELINES
ITEM_PIPELINES
@@ -1371,3 +1388,4 @@ case to see how to enable and use them.
.. _Amazon web services: https://aws.amazon.com/
.. _breadth-first order: https://en.wikipedia.org/wiki/Breadth-first_search
.. _depth-first order: https://en.wikipedia.org/wiki/Depth-first_search
+.. _Google Cloud Storage: https://cloud.google.com/storage/
\ No newline at end of file
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 5d2862980..c17e94a64 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -168,6 +168,8 @@ FTP_USER = 'anonymous'
FTP_PASSWORD = 'guest'
FTP_PASSIVE_MODE = True
+GCS_PROJECT_ID = None
+
HTTPCACHE_ENABLED = False
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_MISSING = False