Scrapy, a fast high-level web crawling & scraping framework for Python.
Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600)
Adrianc99f6e22094e
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 11c2656da..b3c58d6d3 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -387,6 +387,26 @@ crawl::curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3+.. _large-project-startup:++Reducing startup time in large projects+=======================================++When running a spider with ``scrapy crawl``, Scrapy loads all modules listed in+:setting:`SPIDER_MODULES` to find the target spider. In large projects with+many spiders, this can noticeably increase startup time and memory usage.++To avoid loading every spider module, override :setting:`SPIDER_MODULES` on the+command line to point only to the module that contains the spider you want to+run:++.. code-block:: shell++ scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider++Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple+modules by separating them with commas.+.. _bans:Avoiding getting banned