Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

More documentation for setting spider atributes

Shinichi Takayanagibd60c3f41fd2
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index e772a461c..26c247cdd 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -135,6 +135,9 @@ Here are some examples to illustrate:
- ``s3://mybucket/scraping/feeds/%(name)s/%(time)s.json``
+.. note:: :ref:`Spider arguments <spiderargs>` become spider attributes, hence
+ they can also be used as storage URI parameters.
+
.. _topics-feed-storage-backends:
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index 2056664c7..a3e9f410f 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -294,6 +294,14 @@ The above example can also be written as follows::
def start_requests(self):
yield scrapy.Request(f'http://www.example.com/categories/{self.category}')
+If you are :ref:`running Scrapy from a script <run-from-script>`, you can
+specify spider arguments when calling
+:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
+:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`::
+
+ process = CrawlerProcess()
+ process.crawl(MySpider, category="electronics")
+
Keep in mind that spider arguments are only strings.
The spider will not do any parsing on its own.
If you were to set the ``start_urls`` attribute from the command line,