Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600)

Adrianc99f6e22094e
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 11c2656da..b3c58d6d3 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -387,6 +387,26 @@ crawl::
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
+.. _large-project-startup:
+
+Reducing startup time in large projects
+=======================================
+
+When running a spider with ``scrapy crawl``, Scrapy loads all modules listed in
+:setting:`SPIDER_MODULES` to find the target spider. In large projects with
+many spiders, this can noticeably increase startup time and memory usage.
+
+To avoid loading every spider module, override :setting:`SPIDER_MODULES` on the
+command line to point only to the module that contains the spider you want to
+run:
+
+.. code-block:: shell
+
+ scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider
+
+Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple
+modules by separating them with commas.
+
.. _bans:
Avoiding getting banned