Scrapy, a fast high-level web crawling & scraping framework for Python.
Document “logging settings” as special settings (#7668)
Adrianc690eac770a9
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rstindex 8a04f7ada..e7faf48c4 100644--- a/docs/topics/practices.rst+++ b/docs/topics/practices.rst@@ -347,10 +347,10 @@ finishes before starting the next one:install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")react(deferred_f_from_coro_f(crawl))-.. note:: When running multiple spiders in the same process, :ref:`reactor- settings <reactor-settings>` should not have a different value per spider.- Also, :ref:`pre-crawler settings <pre-crawler-settings>` cannot be defined- per spider.+.. note:: When running multiple spiders in the same process, :ref:`logging+ settings <logging-settings>` and :ref:`reactor settings <reactor-settings>`+ should not have a different value per spider, and :ref:`pre-crawler+ settings <pre-crawler-settings>` cannot be defined per spider... seealso:: :ref:`run-from-script`.diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 455985a56..c452f88d1 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -361,6 +361,31 @@ All of these settings, except for :setting:`ASYNCIO_EVENT_LOOP`, are only usedwhen the Twisted reactor is used, i.e. when :setting:`TWISTED_REACTOR_ENABLED`is ``True``.+.. _logging-settings:++Logging settings+----------------++**Logging settings** are settings that configure the global root logging+handler installed by :func:`~scrapy.utils.log.configure_logging`.++These settings can be defined from a spider. However, because only 1 root+logging handler is active per process, these settings cannot use a different+value per spider when :ref:`running multiple spiders in the same process+<run-multiple-spiders>`.++These settings are:++- :setting:`LOG_DATEFORMAT`+- :setting:`LOG_ENABLED`+- :setting:`LOG_ENCODING`+- :setting:`LOG_FILE`+- :setting:`LOG_FILE_APPEND`+- :setting:`LOG_FORMAT`+- :setting:`LOG_LEVEL`+- :setting:`LOG_SHORT_NAMES`+- :setting:`LOG_STDOUT`+.. _topics-settings-ref:Built-in settings reference@@ -481,6 +506,8 @@ Note that the event loop class must inherit from :class:`asyncio.AbstractEventLo:func:`asyncio.set_event_loop`, which will set the specified event loopas the current loop for the current OS thread.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: BOT_NAMEBOT_NAME@@ -662,6 +689,8 @@ Whether to enable DNS in-memory cache.:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effecteither when :setting:`DNS_RESOLVER` is set to a different resolver.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: DNSCACHE_SIZEDNSCACHE_SIZE@@ -671,6 +700,8 @@ Default: ``10000``DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: TWISTED_DNS_RESOLVERTWISTED_DNS_RESOLVER@@ -688,6 +719,8 @@ take the :setting:`DNS_TIMEOUT` setting into account... note::This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: DNS_TIMEOUTDNS_TIMEOUT@@ -703,6 +736,8 @@ Timeout for processing of DNS queries in seconds. Float is supported.:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effecteither when :setting:`DNS_RESOLVER` is set to a different resolver.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: DOWNLOADERDOWNLOADER@@ -1442,6 +1477,8 @@ Default: ``True``Whether to enable logging.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_ENCODINGLOG_ENCODING@@ -1451,6 +1488,8 @@ Default: ``'utf-8'``The encoding to use for logging.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_FILELOG_FILE@@ -1460,6 +1499,8 @@ Default: ``None``File name to use for logging output. If ``None``, standard error will be used.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_FILE_APPENDLOG_FILE_APPEND@@ -1470,6 +1511,8 @@ Default: ``True``If ``False``, the log file specified with :setting:`LOG_FILE` will beoverwritten (discarding the output from previous runs, if any).+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_FORMATLOG_FORMAT@@ -1481,6 +1524,8 @@ String for formatting log messages. Refer to the:ref:`Python logging documentation <logrecord-attributes>` for the wholelist of available placeholders.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_DATEFORMATLOG_DATEFORMAT@@ -1493,6 +1538,8 @@ in :setting:`LOG_FORMAT`. Refer to the:ref:`Python datetime documentation <strftime-strptime-behavior>` for thewhole list of available directives.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_FORMATTERLOG_FORMATTER@@ -1512,6 +1559,8 @@ Default: ``'DEBUG'``Minimum level to log. Available levels are: CRITICAL, ERROR, WARNING,INFO, DEBUG. For more info see :ref:`topics-logging`.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_STDOUTLOG_STDOUT@@ -1523,6 +1572,8 @@ If ``True``, all standard output (and error) of your process will be redirectedto the log. For example if you ``print('hello')`` it will appear in the Scrapylog.+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_SHORT_NAMESLOG_SHORT_NAMES@@ -1533,6 +1584,8 @@ Default: ``False``If ``True``, the logs will just contain the root path. If it is set to ``False``then it displays the component responsible for the log output+.. note:: This is a :ref:`logging setting <logging-settings>`.+.. setting:: LOG_VERSIONSLOG_VERSIONS@@ -1695,6 +1748,8 @@ multi-purpose thread pool used by various Scrapy components. ThreadedDNS Resolver, BlockingFeedStorage, S3FilesStore just to name a few. Increasethis value if you're experiencing problems with insufficient blocking IO.+.. note:: This is a :ref:`reactor setting <reactor-settings>`.+.. setting:: REDIRECT_PRIORITY_ADJUSTREDIRECT_PRIORITY_ADJUST@@ -1921,6 +1976,8 @@ Default: ``'scrapy.spiderloader.SpiderLoader'``The class that will be used for loading spiders, which must implement the:ref:`topics-api-spiderloader`.+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.+.. setting:: SPIDER_LOADER_WARN_ONLYSPIDER_LOADER_WARN_ONLY@@ -1933,6 +1990,8 @@ it will fail loudly if there is any ``ImportError`` or ``SyntaxError`` exceptionBut you can choose to silence this exception and turn it into a simplewarning by setting ``SPIDER_LOADER_WARN_ONLY = True``.+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.+.. setting:: SPIDER_MIDDLEWARESSPIDER_MIDDLEWARES@@ -1978,6 +2037,8 @@ Example:SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"]+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.+.. setting:: STATS_CLASSSTATS_CLASS@@ -2049,7 +2110,7 @@ stopped) will not apply. This mode is currently experimental and may not besuitable for production use. It may also not be supported by 3rd-party code.See :ref:`asyncio-without-reactor` for more information about this mode.-.. note:: This setting can't be set :ref:`per-spider <spider-settings>`.+.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`... versionadded:: 2.15.0@@ -2156,6 +2217,7 @@ current platform.For additional information, see :doc:`core/howto/choosing-reactor`.+.. note:: This is a :ref:`reactor setting <reactor-settings>`... setting:: URLLENGTH_LIMIT