Scrapy, a fast high-level web crawling & scraping framework for Python.
DOC tweak release notes: promote response.follow, mention logging/stats changes
Mikhail Korobovedcde7a2cf1b
diff --git a/docs/news.rst b/docs/news.rstindex 7debe176b..b5e31e444 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -11,28 +11,41 @@ but quite a few handy improvements nonetheless.Scrapy now supports anonymous FTP sessions with customizable user andpassword via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings.-**And if you're using Twisted version 17.1.0 or above, FTP is now available-with Python 3.**--Link extractors now work similarly to what a regular modern browser would-do. Especially, leading and trailing whitespace are removed from attributes-(think ``href=" http://example.com"``) when building ``Link`` objects.-This whitespace-stripping also happens for ``action`` attributes with-``FormRequest``.-**Please also note that link extractors do not canonicalize URLs by default-anymore.** This was puzzling users every now and then, and it's not what-browsers do in fact, so we removed that extra transformation on extractred-links.+And if you're using Twisted version 17.1.0 or above, FTP is now available+with Python 3.++There's a new :meth:`response.follow <scrapy.http.TextResponse.follow>` method+for creating requests; **it is now a recommended way to create Requests+in Scrapy spiders**. This method makes it easier to write correct+spiders; ``response.follow`` has several advantages over creating+``scrapy.Request`` objects directly:++* it handles relative URLs;+* it works properly with non-ascii URLs on non-UTF8 pages;+* in addition to absolute and relative URLs it supports Selectors;+ for ``<a>`` elements it can also extract their href values.++For example, instead of this::-There's a new ``response.follow()`` shortcut for creating requests directly-from a response instance and a relative URL.-For example, instead of::+ for href in response.css('li.page a::attr(href)').extract():+ url = response.urljoin(href)+ yield scrapy.Request(url, self.parse, encoding=response.encoding)- scrapy.Request(response.urljoin(somehrefvalue))+One can now write this::-you can now use the simpler::+ for a in response.css('li.page a'):+ yield response.follow(a, self.parse)- response.follow(somehrefvalue)+Link extractors are also improved. They work similarly to what a regular+modern browser would do: leading and trailing whitespace are removed+from attributes (think ``href=" http://example.com"``) when building+``Link`` objects. This whitespace-stripping also happens for ``action``+attributes with ``FormRequest``.++**Please also note that link extractors do not canonicalize URLs by default+anymore.** This was puzzling users every now and then, and it's not what+browsers do in fact, so we removed that extra transformation on extractred+links.For those of you wanting more control on the ``Referer:`` header that Scrapysends when following links, you can set your own ``Referrer Policy``.@@ -44,6 +57,10 @@ And this policy is fully customizable with W3C standard values(or with something really custom of your own if you wish).See :setting:`REFERRER_POLICY` for details.+To make Scrapy spiders easier to debug, Scrapy logs more stats by default+in 1.4: memory usage stats, detailed retry stats, detailed HTTP error code+stats. A similar change is that HTTP cache path is also visible in logs now.+Last but not least, Scrapy now has the option to make JSON and XML itemsmore human-readable, with newlines between items and even custom indentingoffset, using the new :setting:`FEED_EXPORT_INDENT` setting.@@ -60,7 +77,7 @@ New Features- Enable memusage extension by default (:issue:`2187`) ;**this is technically backwards-incompatible** so please check if you haveany non-default ``MEMUSAGE_***`` settings set.-- New :meth:`Response.follow <scrapy.http.Response.follow>` shortcut+- New :ref:`response.follow <response-follow-example>` shortcutfor creating requests (:issue:`1940`)- Added ``flags`` argument and attribute to :class:`Request <scrapy.http.Request>`objects (:issue:`2047`)