Scrapy, a fast high-level web crawling & scraping framework for Python.
DOC how to override log level for a specific Scrapy component
Mikhail Korobovda19f0b7b73c
diff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex 231f5186b..ac3b614fc 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -10,7 +10,7 @@ Loggingabout the new logging system.Scrapy uses `Python's builtin logging system-<https://docs.python.org/2/library/logging.html>`_ for event logging. We'll+<https://docs.python.org/3/library/logging.html>`_ for event logging. We'llprovide some simple examples to get you started, but for more advanceduse-cases it's strongly suggested to read thoroughly its documentation.@@ -193,6 +193,43 @@ to override some of the Scrapy settings regarding logging.Module `logging.handlers <https://docs.python.org/2/library/logging.handlers.html>`_Further documentation on available handlers+Advanced customization+----------------------++Because Scrapy uses stdlib logging module, you can customize logging using+all features of stdlib logging.++For example, let's say you're scraping a website which returns many+HTTP 404 and 500 responses, and you want to hide all messages like this::++ 2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring+ response <500 http://quotes.toscrape.com/page/1-34/>: HTTP status code+ is not handled or not allowed++The first thing to note is a logger name - it is in brackets:+``[scrapy.spidermiddlewares.httperror]``. If you get just ``[scrapy]`` then+:setting:`LOG_SHORT_NAMES` is likely set to True; set it to False and re-run+the crawl.++Next, we can see that the message has INFO level. To hide it+we should set logging level for ``scrapy.spidermiddlewares.httperror``+higher than INFO; next level after INFO is WARNING. It could be done+e.g. in the spider's ``__init__`` method::++ import logging+ import scrapy+++ class MySpider(scrapy.Spider):+ # ...+ def __init__(self, *args, **kwargs):+ logger = logging.getLogger('scrapy.spidermiddlewares.httperror')+ logger.setLevel(logging.WARNING)+ super().__init__(*args, **kwargs)++If you run this spider again then INFO messages from+``scrapy.spidermiddlewares.httperror`` logger will be gone.+scrapy.utils.log module=======================