Scrapy, a fast high-level web crawling & scraping framework for Python.
applying review comments
Elias Dorneles89a088b3c349
diff --git a/docs/news.rst b/docs/news.rstindex 775513a98..0c5228ca2 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -12,7 +12,7 @@ This 1.1 release brings a lot of interesting features and bug fixes::ref:`news_basicpy3` for more details and some limitations.- Hot new features:- - ItemLoaders now support nested loaders (:issue:`1467`).+ - Item loaders now support nested loaders (:issue:`1467`).- ``FormRequest.from_response`` improvements (:issue:`1382`, :issue:`1137`).- Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improvedAutoThrottle docs (:issue:`1324`).@@ -20,38 +20,47 @@ This 1.1 release brings a lot of interesting features and bug fixes:- Anonymous S3 connections (:issue:`1358`).- Deferreds in downloader middlewares (:issue:`1473`). This enables betterrobots.txt handling (:issue:`1471`).- - HTTP cache improvements (:issue:`1151`).+ - HTTP caching now follows RFC2616 more closely, added settings+ :setting:`HTTPCACHE_ALWAYS_STORE` and+ :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:issue:`1151`).- These bug fixes may require your attention:- - Don't retry bad requests (HTTP 400) (:issue:`1289`).+ - Don't retry bad requests (HTTP 400) by default (:issue:`1289`).+ If you need the old behavior, add ``400`` to :setting:`RETRY_HTTP_CODES`.- Fix shell files argument handling (:issue:`1710`, :issue:`1550`).- - Fixes on robots.txt handling (:issue:`1783`).- - Exporters work on unicode (:issue:`1080`).- - Fix xmliter to accept nodenames with dots (:issue:`1533`).+ If you try ``scrapy shell index.html`` it will try to load the URL http://index.html,+ use ``scrapy shell ./index.html`` to load a local file.+ - Robots.txt compliance is now enabled by default for newly-created projects+ (:issue:`1724`).+ - Exporters now work on unicode, instead of bytes by default (:issue:`1080`).+ If you use ``PythonItemExporter``, you may want to update your code to+ disable binary mode which is now deprecated.+ - Accept XML node names containing dots as valid (:issue:`1533`).Keep reading for more details on other improvements and bug fixes... _news_basicpy3:-Basic Python 3 Support-~~~~~~~~~~~~~~~~~~~~~~+Beta Python 3 Support+~~~~~~~~~~~~~~~~~~~~~-We have been hard at work to make Scrapy run on Python 3. As a result, now you-can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some+We have been `hard at work to make Scrapy run on Python 3+<https://github.com/scrapy/scrapy/wiki/Python-3-Porting>`_. As a result, now+you can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Somefeatures are still missing (and some may never be ported).+Almost all addons/middlewares are expected to work. However, we are aware of-some limitations:+some limitations in Python 3:+- Doesn't work in Windows yet (non-Python 3 ported Twisted dependency)- S3 downloads are not supported (see :issue:`1718`)-- Sending emails is not supported+- Sending emails is not supported (non-Python 3 ported Twisted dependency)- FTP download handler is not supported (non-Python 3 ported Twisteddependency)- Telnet is not supported (non-Python 3 ported Twisted dependency)- There are problems with non-ASCII URLs in Python 3-- Reported problems with HTTP caches created by Scrapy in Python 2.x which- can't be reused in Scrapy in Python 3.x (to be checked)- There is also a nasty issue with `cryptography` library: recent versionsdon't work well on OS X + Python 3.5(see https://github.com/pyca/cryptography/issues/2690). As a workaround, you@@ -72,9 +81,6 @@ Additional New Features and Enhancements- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default checkinterval (:issue:`1282`).-- HTTP caching now follows RFC2616 more closely, added settings- :setting:`HTTPCACHE_ALWAYS_STORE` and- :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:issue:`1151`).- Download handlers are now lazy-loaded on first request using theirscheme (:issue:`1390`, :issue:`1421`).- ``RedirectMiddleware`` now skips the status codes from@@ -87,10 +93,8 @@ Additional New Features and Enhancements- an empty string is now used for submit buttons without a value(:issue:`1472`)-- Item Loaders now support nested loaders (:issue:`1467`).- Dict-like settings now have per-key priorities(:issue:`1135`, :issue:`1149` and :issue:`1586`).-- robots.txt compliance now enabled by default for new projects (:issue:`1724`).- ``CloseSpider`` and ``SpiderState`` extensions now get disabled if no relevantsetting is set (:issue:`1723`, :issue:`1725`).- Added method ``ExecutionEngine.close`` (:issue:`1423`).