Scrapy, a fast high-level web crawling & scraping framework for Python.
Enable robots.txt handling by default for new projects. Fixes GH-1668.
Mikhail Korobovf30758c246ef
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex cc070d8c0..0959a87a7 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -750,8 +750,8 @@ Default: ``60.0``Scope: ``scrapy.extensions.memusage``The :ref:`Memory usage extension <topics-extensions-ref-memusage>`-checks the current memory usage, versus the limits set by-:setting:`MEMUSAGE_LIMIT_MB` and :setting:`MEMUSAGE_WARNING_MB`,+checks the current memory usage, versus the limits set by+:setting:`MEMUSAGE_LIMIT_MB` and :setting:`MEMUSAGE_WARNING_MB`,at fixed time intervals.This sets the length of these intervals, in seconds.@@ -877,7 +877,13 @@ Default: ``False``Scope: ``scrapy.downloadermiddlewares.robotstxt``If enabled, Scrapy will respect robots.txt policies. For more information see-:ref:`topics-dlmw-robots`+:ref:`topics-dlmw-robots`.++.. note::++ While the default value is ``False`` for historical reasons,+ this option is enabled by default in settings.py file generated+ by ``scrapy startproject`` command... setting:: SCHEDULER@@ -1036,7 +1042,7 @@ TEMPLATES_DIRDefault: ``templates`` dir inside scrapy moduleThe directory where to look for templates when creating new projects with-:command:`startproject` command and new spiders with :command:`genspider`+:command:`startproject` command and new spiders with :command:`genspider`command.The project name must not conflict with the name of custom files or directoriesdiff --git a/scrapy/templates/project/module/settings.py.tmpl b/scrapy/templates/project/module/settings.py.tmplindex 822812c9a..f13e85871 100644--- a/scrapy/templates/project/module/settings.py.tmpl+++ b/scrapy/templates/project/module/settings.py.tmpl@@ -18,6 +18,9 @@ NEWSPIDER_MODULE = '$project_name.spiders'# Crawl responsibly by identifying yourself (and your website) on the user-agent#USER_AGENT = '$project_name (+http://www.yourdomain.com)'+# Obey robots.txt rules+ROBOTSTXT_OBEY = True+# Configure maximum concurrent requests performed by Scrapy (default: 16)#CONCURRENT_REQUESTS = 32