Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document additional request metadata keys (#7933)

Adrian7133d115c862
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 0417ce9e1..c45ec0487 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -827,6 +827,19 @@ redirect. For example, :class:`RedirectMiddleware` indicates the triggering
response status code as an integer, while :class:`MetaRefreshMiddleware`
always uses the ``'meta refresh'`` string as reason.
+.. reqmeta:: redirect_times
+
+The number of redirects that a request has gone through so far can be found in
+the ``redirect_times`` :attr:`Request.meta <scrapy.Request.meta>` key.
+
+.. reqmeta:: redirect_ttl
+
+The number of redirects that a request may still go through is tracked in the
+``redirect_ttl`` :attr:`Request.meta <scrapy.Request.meta>` key, which is
+initialized from :setting:`REDIRECT_MAX_TIMES` and decreased on every redirect.
+Set it on a request to allow fewer redirects for that request;
+:setting:`REDIRECT_MAX_TIMES` still applies as an upper bound.
+
The :class:`RedirectMiddleware` can be configured through the following
settings (see the settings documentation for more info):
@@ -984,6 +997,18 @@ RetryMiddleware
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
set to True, the request will be ignored by this middleware.
+.. reqmeta:: retry_times
+
+The number of times that a request has been retried so far can be found in the
+``retry_times`` :attr:`Request.meta <scrapy.Request.meta>` key. Since it counts
+against :setting:`RETRY_TIMES`, copying it into a follow-up request lowers the
+number of retries allowed for that request.
+
+.. reqmeta:: priority_adjust
+
+The ``priority_adjust`` :attr:`Request.meta <scrapy.Request.meta>` key
+overrides :setting:`RETRY_PRIORITY_ADJUST` for a request.
+
To retry requests from a spider callback, you can use the
:func:`get_retry_request` function:
@@ -1088,6 +1113,8 @@ Adjust retry request priority relative to original request:
- a positive priority adjust means higher priority.
- **a negative priority adjust (default) means lower priority.**
+See also: :reqmeta:`priority_adjust`.
+
.. _topics-dlmw-robots:
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 7344292a2..4a6d19ce2 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -161,7 +161,7 @@ Request objects
A common use of request metadata is to define request-specific
parameters for Scrapy components (extensions, middlewares, etc.). For
- example, if you set ``dont_retry`` to ``True``,
+ example, if you set :reqmeta:`dont_retry` to ``True``,
:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` will never
retry that request, even if it fails. See :ref:`topics-request-meta`.
@@ -169,15 +169,15 @@ Request objects
example, to keep request state information relevant to your component.
For example,
:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses the
- ``retry_times`` metadata key to keep track of how many times a request
- has been retried so far.
+ :reqmeta:`retry_times` metadata key to keep track of how many times a
+ request has been retried so far.
Copying all the metadata of a previous request into a new, follow-up
request in a spider callback is a bad practice, because request
metadata may include metadata set by Scrapy components that is not
meant to be copied into other requests. For example, copying the
- ``retry_times`` metadata key into follow-up requests can lower the
- amount of retries allowed for those follow-up requests.
+ :reqmeta:`retry_times` metadata key into follow-up requests can lower
+ the amount of retries allowed for those follow-up requests.
You should only copy all request metadata from one request to another
if the new request is meant to replace the old request, as is often the
@@ -828,6 +828,7 @@ Those are:
* :reqmeta:`autothrottle_dont_adjust_delay`
* :reqmeta:`bindaddress`
* :reqmeta:`cookiejar`
+* :reqmeta:`depth`
* :reqmeta:`dont_cache`
* :reqmeta:`dont_merge_cookies`
* :reqmeta:`dont_obey_robotstxt`
@@ -837,10 +838,12 @@ Those are:
* :reqmeta:`download_latency`
* :reqmeta:`download_maxsize`
* :reqmeta:`download_slot`
-* :reqmeta:`download_warnsize`
* :reqmeta:`download_timeout`
-* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
-* ``ftp_user`` (See :setting:`FTP_USER` for more info)
+* :reqmeta:`download_warnsize`
+* :reqmeta:`ftp_local_filename`
+* :reqmeta:`ftp_passive`
+* :reqmeta:`ftp_password`
+* :reqmeta:`ftp_user`
* :reqmeta:`give_up_log_level`
* :reqmeta:`handle_httpstatus_all`
* :reqmeta:`handle_httpstatus_list`
@@ -848,13 +851,22 @@ Those are:
* :reqmeta:`http_pass`
* :reqmeta:`http_user`
* :reqmeta:`is_start_request`
+* :reqmeta:`link_text`
* :reqmeta:`max_retry_times`
+* :reqmeta:`priority_adjust`
* :reqmeta:`proxy`
* :reqmeta:`redirect_reasons`
+* :reqmeta:`redirect_times`
+* :reqmeta:`redirect_ttl`
* :reqmeta:`redirect_urls`
* :reqmeta:`referrer_policy`
+* :reqmeta:`retry_times`
+* :reqmeta:`rule`
* :reqmeta:`verbatim_url`
+Scrapy components also use meta keys whose name starts with an underscore, such
+as ``_auth_proxy``. Those are internal, and may change or disappear at any time.
+
.. reqmeta:: bindaddress
bindaddress
@@ -921,6 +933,15 @@ download_fail_on_dataloss
Whether or not to fail on broken responses. See:
:setting:`DOWNLOAD_FAIL_ON_DATALOSS`.
+.. reqmeta:: ftp_local_filename
+
+ftp_local_filename
+------------------
+
+Path, as :class:`bytes`, of the file where to write the response body of an
+``ftp://`` request. If set, :attr:`Response.body <scrapy.http.Response.body>`
+holds this path instead of the file contents.
+
.. reqmeta:: give_up_log_level
give_up_log_level
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index f7f5bf8ae..aa4208607 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1085,6 +1085,7 @@ handler (without replacement), place this in your ``settings.py``:
:ref:`security-local-resources`
+.. reqmeta:: download_slot
.. setting:: DOWNLOAD_SLOTS
DOWNLOAD_SLOTS
@@ -1109,6 +1110,14 @@ Allows to define concurrency/delay parameters on per slot (domain) basis:
- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency``
- :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay``
+Requests are assigned to a slot based on their URL domain. To assign a request
+to a specific slot instead, set the name of the slot as the ``download_slot``
+:attr:`Request.meta <scrapy.Request.meta>` key. Once a request is assigned to a
+slot, that key holds the name of the slot.
+
+Since that key is kept on redirects, a redirected request stays in the slot of
+the request it comes from, even when it points to a different domain.
+
.. setting:: DOWNLOAD_TIMEOUT
@@ -1438,6 +1447,7 @@ non-default value in :ref:`per-spider settings <spider-settings>`.
.. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
+.. reqmeta:: ftp_passive
.. setting:: FTP_PASSIVE_MODE
FTP_PASSIVE_MODE
@@ -1445,7 +1455,8 @@ FTP_PASSIVE_MODE
Default: ``True``
-Whether or not to use passive mode when initiating FTP transfers.
+Whether or not to use passive mode when initiating FTP transfers, unless there
+is an ``"ftp_passive"`` key in ``Request`` meta.
.. note::
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 219eff206..cb5608ac6 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -233,6 +233,8 @@ DepthMiddleware
.. module:: scrapy.spidermiddlewares.depth
:synopsis: Depth Spider Middleware
+.. reqmeta:: depth
+
.. autoclass:: DepthMiddleware
HttpErrorMiddleware
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index a932dc72f..ae79ec3ee 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -491,6 +491,15 @@ CrawlSpider
described below. If multiple rules match the same link, the first one
will be used, according to the order they're defined in this attribute.
+ .. reqmeta:: rule
+
+ Requests generated from :attr:`rules` carry the index of the matching rule
+ within :attr:`rules` in their ``rule``
+ :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs
+ that key to dispatch the response to the right rule, so copying it into a
+ request generated by a different rule sends the response to the wrong
+ callback.
+
This spider also exposes an overridable method:
.. method:: parse_start_url(response, **kwargs)
@@ -504,6 +513,8 @@ CrawlSpider
Crawling rules
~~~~~~~~~~~~~~
+.. reqmeta:: link_text
+
.. autoclass:: Rule
``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
diff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.py
index 555d930e6..8008ca9d2 100644
--- a/scrapy/http/cookies.py
+++ b/scrapy/http/cookies.py
@@ -154,14 +154,11 @@ class WrappedRequest:
return urlparse_cached(self.request).scheme
def is_unverifiable(self) -> bool:
- """Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965.
-
- It defaults to False. An unverifiable request is one whose URL the user did not have the
- option to approve. For example, if the request is for an image in an
- HTML document, and the user had no option to approve the automatic
- fetching of the image, this should be true.
+ """Return ``False``, as Scrapy does not track whether the user had the
+ option to approve the URL of a request, which is what makes a request
+ unverifiable as defined by :rfc:`2965`.
"""
- return cast("bool", self.request.meta.get("is_unverifiable", False))
+ return False
@property
def full_url(self) -> str:
diff --git a/tests/test_http_cookies.py b/tests/test_http_cookies.py
index 5d4d5a2bd..8840172a5 100644
--- a/tests/test_http_cookies.py
+++ b/tests/test_http_cookies.py
@@ -79,11 +79,6 @@ class TestWrappedRequest:
assert not self.wrapped.is_unverifiable()
assert not self.wrapped.unverifiable
- def test_is_unverifiable2(self):
- self.request.meta["is_unverifiable"] = True
- assert self.wrapped.is_unverifiable()
- assert self.wrapped.unverifiable
-
def test_get_origin_req_host(self):
assert self.wrapped.origin_req_host == "www.example.com"