Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Rephrase "New features" section

Paul Tremberth8729a91f7a13
diff --git a/docs/news.rst b/docs/news.rst
index 675a6d595..979e9a953 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -9,24 +9,40 @@ Scrapy 1.4.0 (2017-XX-XX)
New Features
~~~~~~~~~~~~
-- Use credentials from request.meta['proxy'] #2530
-- [httpcompression] add support for br - brotli content encoding #2535
-- Enable memusage extension by default. #2539
-- response.follow #2540
-- add flags to request #2082
-- Support Anonymous FTP #2343
-- HttpErrorMiddleware stats #2566
-- Retry stats #2543
-- Set canonicalize=False for LinkExtractor #2537
-- Referrer policies in RefererMiddleware #2306
-- Fix referrer policy from response headers and support explicit empty string #2627
-- Data URI download handler. #2334
-- HttpCacheMiddleware: log cache directory at instantiation #2611
-- Add warning on duplicate spider name #2612
-- Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer #2646
-- Allow redirections in media files downloads #2616
-- Travis CI: use portable pypy for Linux #2710
-
+- Accept proxy credentials in request.meta['proxy'] (:issue:`2526`)
+- Support `brotli`_-compressed content; requires optional `brotlipy`_
+ (:issue:`2535`)
+- Enable memusage extension by default (:issue:`2187`) ;
+ **this is technically backwards-incompatible** so please check if you have
+ any non-default ``MEMUSAGE_***`` settings set.
+- New :meth:`Response.follow <scrapy.http.Response.follow>` shortcur
+ for creating requests (:issue:`1940`)
+- Added ``flags`` argument and attribute to :class:`Request <scrapy.http.Request>`
+ (:issue:`2047`)
+- Support Anonymous FTP (:issue:`2342`)
+- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/***``
+ stats to :class:`RetryMiddleware <scrapy.downloadermiddlewares.retry.RetryMiddleware>`
+ (:issue:`2543`)
+- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/***``
+ stats to :class:`HttpErrorMiddleware <scrapy.spidermiddlewares.httperror.HttpErrorMiddleware>`
+ (:issue:`2566`)
+- Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor`
+ (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`):
+ **warning, this istechnically backwards-incompatible**
+- Customizable :setting:`Referrer policy <REFERRER_POLICY>` in
+ :class:`RefererMiddleware <scrapy.spidermiddlewares.referer.RefererMiddleware>`
+ (:issue:`2306`)
+- New ``data:`` URI download handler (:issue:`2334`, fixes :issue:`2156`)
+- Log cache directory when HTTP Cache is used (:issue:`2611`, fixes :issue:`2604`)
+- Warn users when project contains duplicate spider names (fixes :issue:`2181`)
+- :class:`CaselessDict` now accepts ``Mapping`` instances and not only dicts (:issue:`2646`)
+- :ref:`Media downloads <topics-media-pipeline>`, with :class:`FilesPipelines`
+ or :class:`ImagesPipelines`, can now optionally handle HTTP redirects
+ using the new :setting:`MEDIA_ALLOW_REDIRECTS` (:issue:`2616`, fixes :issue:`2004`)
+- Use portable pypy for Linux on Travis CI (:issue:`2710`)
+
+.. _brotli: https://github.com/google/brotli
+.. _brotlipy: https://github.com/python-hyper/brotlipy/
Bug fixes
~~~~~~~~~
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index f258ff748..e948913a4 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -320,8 +320,6 @@ all be dropped because at least one dimension is shorter than the constraint.
By default, there are no size constraints, so all images are processed.
-.. _topics-media-pipeline-override:
-
Allowing redirections
---------------------
@@ -330,10 +328,11 @@ Allowing redirections
By default media pipelines ignore redirects, i.e. an HTTP redirection
to a media file URL request will mean the media download is considered failed.
-To handle media redirections, set this settings to ``True``:
+To handle media redirections, set this setting to ``True``::
MEDIA_ALLOW_REDIRECTS = True
+.. _topics-media-pipeline-override:
Extending the Media Pipelines
=============================
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index f1552572a..6ca37b7c9 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -24,7 +24,7 @@ below in :ref:`topics-request-response-ref-request-subclasses` and
Request objects
===============
-.. class:: Request(url[, callback, method='GET', headers, body, cookies, meta, encoding='utf-8', priority=0, dont_filter=False, errback])
+.. class:: Request(url[, callback, method='GET', headers, body, cookies, meta, encoding='utf-8', priority=0, dont_filter=False, errback, flags])
A :class:`Request` object represents an HTTP request, which is usually
generated in the Spider and executed by the Downloader, and thus generating