Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Update docs for new "referrer_policy" Request.meta key

Paul Tremberth03ff19d1882e
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 3d110b02d..67f8ec285 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -307,6 +307,7 @@ Those are:
* :reqmeta:`proxy`
* ``ftp_user`` (See :setting:`FTP_USER` for more info)
* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
+* :reqmeta:`referrer_policy`
.. reqmeta:: bindaddress
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 0ddf027ea..a4ac45b41 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -337,6 +337,8 @@ REFERER_POLICY
Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'``
+.. reqmeta:: referrer_policy
+
`Referrer Policy`_ to apply when populating Request "Referer" header.
This setting accepts:
@@ -370,6 +372,11 @@ using ``file://`` or ``s3://`` scheme.
``same-origin`` may be a better choice if you want to remove referrer
information for cross-domain requests.
+.. note::
+ You can also override the Referrer Policy per request,
+ using the special ``"referrer_policy"`` :ref:`Request.meta <topics-request-meta>` key,
+ with the same acceptable values as for the ``REFERER_POLICY`` setting.
+
.. _Referrer Policy: https://www.w3.org/TR/referrer-policy
.. _"no-referrer": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer
.. _"no-referrer-when-downgrade": https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade