Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Improve pre-crawler setting docs (#7835)

Adrian434fd1154ad2
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index ee2c3a3cd..e8a843e80 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -665,6 +665,8 @@ Example:
COMMANDS_MODULE = "mybot.commands"
+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
+
.. _Deploying your project: https://scrapyd.readthedocs.io/en/latest/deploy.html
Register commands via setup.py entry points
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 1b6851d04..81055afc2 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -305,10 +305,21 @@ These settings cannot be :ref:`set from a spider <spider-settings>`.
These settings are:
-- :setting:`TWISTED_REACTOR_ENABLED`
+- :setting:`ADDONS`
+- :setting:`COMMANDS_MODULE`
+- :setting:`FORCE_CRAWLER_PROCESS`
- :setting:`SPIDER_LOADER_CLASS` and settings used by the corresponding
spider loader class, e.g. :setting:`SPIDER_MODULES` and
:setting:`SPIDER_LOADER_WARN_ONLY` for the default spider loader class.
+- :setting:`TWISTED_REACTOR_ENABLED`
+
+:setting:`ADDONS` is a special case: it can be set from a spider, but the
+``update_pre_crawler_settings()`` method of :ref:`add-ons <topics-addons>`
+enabled that way is not called.
+
+:setting:`TWISTED_REACTOR` also acts as a pre-crawler setting when running a
+:ref:`command that needs a CrawlerProcess <topics-commands-crawlerprocess>`,
+since its project-level value determines the crawler process class.
.. _reactor-settings:
@@ -409,6 +420,9 @@ Default: ``{}``
A dict containing paths to the add-ons enabled in your project and their
priorities. For more information, see :ref:`topics-addons`.
+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`, with a
+ caveat described in that section.
+
.. setting:: ASYNCIO_EVENT_LOOP
ASYNCIO_EVENT_LOOP
@@ -1402,6 +1416,8 @@ When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``,
Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a
non-default value in :ref:`per-spider settings <spider-settings>`.
+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
+
.. setting:: FTP_PASSIVE_MODE
FTP_PASSIVE_MODE