Scrapy, a fast high-level web crawling & scraping framework for Python.
Adding documentation about Google Cloud Storage Feed Export
Júlio César Batistacb5f800b0f70
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex b64dbfbfd..efb63b0ba 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -185,6 +185,27 @@ passed through the following settings:* :setting:`AWS_ACCESS_KEY_ID`* :setting:`AWS_SECRET_ACCESS_KEY`+.. _topics-feed-storage-gcs:++Google Cloud Storage (GCS)+--------------------------++The feeds are stored on `Google Cloud Storage`_.++ * URI scheme: ``gcs``+ * Example URIs:++ * ``gcs://mybucket/path/to/export.csv``++ * Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.++For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.++You can set a *Project ID* and *Access Control List (ACL)* through the following settings:++ * :setting:`FEED_STORAGE_GCS_ACL`+ * :setting:`GCS_PROJECT_ID`+.. _topics-feed-storage-stdout:Standard output@@ -366,3 +387,4 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter.. _Amazon S3: https://aws.amazon.com/s3/.. _boto: https://github.com/boto/boto.. _botocore: https://github.com/boto/botocore+.. _Google Cloud Storage: https://cloud.google.com/storage/\ No newline at end of filediff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 0ac26a9bd..90ae8fd93 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -749,6 +749,14 @@ The Feed Temp dir allows you to set a custom folder to save crawlertemporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and:ref:`Amazon S3 <topics-feed-storage-s3>`.+.. setting:: FEED_STORAGE_GCS_ACL++FEED_STORAGE_GCS_ACL+--------------------++The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.+For more information on how to set this value, please refer to `Google Cloud documentation <https://cloud.google.com/storage/docs/access-control/lists>`_.+.. setting:: FTP_PASSIVE_MODEFTP_PASSIVE_MODE@@ -786,6 +794,15 @@ Default: ``"anonymous"``The username to use for FTP connections when there is no ``"ftp_user"``in ``Request`` meta.+.. setting:: GCS_PROJECT_ID++GCS_PROJECT_ID+-----------------++Default: ``None``++The Project ID that will be used when storing data on `Google Cloud Storage`_.+.. setting:: ITEM_PIPELINESITEM_PIPELINES@@ -1371,3 +1388,4 @@ case to see how to enable and use them... _Amazon web services: https://aws.amazon.com/.. _breadth-first order: https://en.wikipedia.org/wiki/Breadth-first_search.. _depth-first order: https://en.wikipedia.org/wiki/Depth-first_search+.. _Google Cloud Storage: https://cloud.google.com/storage/\ No newline at end of filediff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.pyindex 5d2862980..c17e94a64 100644--- a/scrapy/settings/default_settings.py+++ b/scrapy/settings/default_settings.py@@ -168,6 +168,8 @@ FTP_USER = 'anonymous'FTP_PASSWORD = 'guest'FTP_PASSIVE_MODE = True+GCS_PROJECT_ID = None+HTTPCACHE_ENABLED = FalseHTTPCACHE_DIR = 'httpcache'HTTPCACHE_IGNORE_MISSING = False