Scrapy, a fast high-level web crawling & scraping framework for Python.
Update docs
Paul Tremberthe249abc32bcb
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex e8325d7ef..0ddf027ea 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -333,7 +333,7 @@ Whether to enable referer middleware.REFERER_POLICY^^^^^^^^^^^^^^-.. versionadded:: 1.3+.. versionadded:: 1.4Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``@@ -346,14 +346,14 @@ This setting accepts:(see ``scrapy.spidermiddlewares.referer``),- or one of the standard W3C-defined string values:- - `"no-referrer" <https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer>`_,- - `"no-referrer-when-downgrade" <https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade>`_,- - `"same-origin" <https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin>`_,- - `"origin" <https://www.w3.org/TR/referrer-policy/#referrer-policy-origin>`_,- - `"strict-origin" <https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin>`_,- - `"origin-when-cross-origin" <https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin>`_,- - `"strict-origin-when-cross-origin" <https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin>`_,- - or `"unsafe-url" <https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url>`_+ - `"no-referrer"`_,+ - `"no-referrer-when-downgrade"`_,+ - `"same-origin"`_,+ - `"origin"`_,+ - `"strict-origin"`_,+ - `"origin-when-cross-origin"`_,+ - `"strict-origin-when-cross-origin"`_,+ - or `"unsafe-url"`_(not recommended).(It can also be the non-standard value ``"scrapy-default"`` to use@@ -364,14 +364,22 @@ with the addition that "Referrer" is not sent if the parent request wasusing ``file://`` or ``s3://`` scheme... warning::- By default, Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,+ Scrapy's default referrer policy, just like `"no-referrer-when-downgrade"`_,will send a non-empty "Referer" header from any ``https://`` to any ``https://`` URL,even if the domain is different.``same-origin`` may be a better choice if you want to remove referrerinformation for cross-domain requests... _Referrer Policy: https://www.w3.org/TR/referrer-policy+.. _"no-referrer": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer.. _"no-referrer-when-downgrade": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade+.. _"same-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin+.. _"origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-origin+.. _"strict-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin+.. _"origin-when-cross-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin+.. _"strict-origin-when-cross-origin": https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin+.. _"unsafe-url": https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url+UrlLengthMiddleware-------------------