Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: document daily log file rotation (#7501)
Abhinav W3b34ab88c071
diff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex a398d6c83..bbb5d0458 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -194,6 +194,48 @@ If :setting:`LOG_SHORT_NAMES` is set, then the logs will not display the Scrapycomponent that prints the log. It is unset by default, hence logs contain theScrapy component responsible for that log output.+Rotating log files+------------------++Scrapy's :setting:`LOG_FILE` setting writes logs to a single file. It does not+rotate log files automatically, but you can use Python's standard+:mod:`logging.handlers` module when running Scrapy from a script.++For example, to rotate the log file every day:++.. skip: next++.. code-block:: python++ import logging+ from logging.handlers import TimedRotatingFileHandler++ from scrapy.crawler import CrawlerProcess+ from scrapy.utils.project import get_project_settings++ from myproject.spiders.myspider import MySpider++ settings = get_project_settings()+ process = CrawlerProcess(settings, install_root_handler=False)++ handler = TimedRotatingFileHandler(+ "scrapy.log",+ when="midnight",+ backupCount=7,+ encoding=settings.get("LOG_ENCODING"),+ )+ handler.setFormatter(+ logging.Formatter(settings.get("LOG_FORMAT"), settings.get("LOG_DATEFORMAT"))+ )++ root_logger = logging.getLogger()+ root_logger.setLevel(settings.get("LOG_LEVEL"))+ root_logger.addHandler(handler)++ process.crawl(MySpider)+ process.start()++Command-line options--------------------