Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add verbose introduction to new features

Paul Tremberthc6464cc4f5a1
diff --git a/docs/news.rst b/docs/news.rst
index a19d7a93a..f225fef1d 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -6,6 +6,52 @@ Release notes
Scrapy 1.4.0 (2017-XX-XX)
-------------------------
+Scrapy 1.4 does not bring that many breathtaking new features
+but quite a few handy improvements nonetheless.
+
+Scrapy now supports anonymous FTP sessions with customizable user and
+password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings.
+**And if you're using Twisted version 17.1.0 or above, FTP is now available
+with Python 3.**
+
+Link extractors now work similarly to what a regular modern browser would
+do. Especially, leading and trailing whitespace are removed from attributes
+(think ``href=" http://example.com"``) when building ``Link`` objects.
+This whitespace-stripping also happens for ``action`` attributes with
+``FormRequest``.
+**Please also note that link extractors do not canonicalize URLs by default
+anymore.** This was puzzling users every now and then, and it's not what
+browsers do in fact, so we removed that extra transformation on extractred
+links.
+
+There's a new ``response.follow()`` shortcut for creating URLs directly
+from the response instance in callbacks.
+For example, instead of::
+
+ scrapy.Request(response.urljoin(somehrefvalue))
+
+you can now use the simpler::
+
+ response.follow(somehrefvalue)
+
+
+For those of you wanting more control on the ``Referer:`` header that Scrapy
+sends when following links, you can set your own ``Referrer Policy``.
+Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and
+blindly set it to the URL of the response that generated the HTTP request
+(which could leak information on your URL seeds).
+By default, Scrapy now behaves much like your regular browser does.
+And this policy is fully customizable with W3C standard values
+(or with something really custom of your own if you wish).
+See :setting:`REFERRER_POLICY` for details.
+
+Last but not least, Scrapy now has the option to make JSON and XML items
+more human-readable, with newlines between items and even custom indenting
+offset, using the new :setting:`FEED_EXPORT_INDENT` setting.
+
+Enjoy! (Or read on for the rest of changes in this release.)
+
+
New Features
~~~~~~~~~~~~