Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

added note about how to revert to old behavior for robotstxt

Elias Dorneles2f697ce59f85
diff --git a/docs/news.rst b/docs/news.rst
index 0c5228ca2..ea797c1ee 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -32,7 +32,8 @@ This 1.1 release brings a lot of interesting features and bug fixes:
If you try ``scrapy shell index.html`` it will try to load the URL http://index.html,
use ``scrapy shell ./index.html`` to load a local file.
- Robots.txt compliance is now enabled by default for newly-created projects
- (:issue:`1724`).
+ (:issue:`1724`). If you need old behavior, update :setting:`ROBOTSTXT_OBEY`
+ in ``settings.py`` file when creating a new project.
- Exporters now work on unicode, instead of bytes by default (:issue:`1080`).
If you use ``PythonItemExporter``, you may want to update your code to
disable binary mode which is now deprecated.