Scrapy, a fast high-level web crawling & scraping framework for Python.
Document additional request metadata keys (#7933)
Adrian7133d115c862
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 0417ce9e1..c45ec0487 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -827,6 +827,19 @@ redirect. For example, :class:`RedirectMiddleware` indicates the triggeringresponse status code as an integer, while :class:`MetaRefreshMiddleware`always uses the ``'meta refresh'`` string as reason.+.. reqmeta:: redirect_times++The number of redirects that a request has gone through so far can be found in+the ``redirect_times`` :attr:`Request.meta <scrapy.Request.meta>` key.++.. reqmeta:: redirect_ttl++The number of redirects that a request may still go through is tracked in the+``redirect_ttl`` :attr:`Request.meta <scrapy.Request.meta>` key, which is+initialized from :setting:`REDIRECT_MAX_TIMES` and decreased on every redirect.+Set it on a request to allow fewer redirects for that request;+:setting:`REDIRECT_MAX_TIMES` still applies as an upper bound.+The :class:`RedirectMiddleware` can be configured through the followingsettings (see the settings documentation for more info):@@ -984,6 +997,18 @@ RetryMiddlewareIf :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` keyset to True, the request will be ignored by this middleware.+.. reqmeta:: retry_times++The number of times that a request has been retried so far can be found in the+``retry_times`` :attr:`Request.meta <scrapy.Request.meta>` key. Since it counts+against :setting:`RETRY_TIMES`, copying it into a follow-up request lowers the+number of retries allowed for that request.++.. reqmeta:: priority_adjust++The ``priority_adjust`` :attr:`Request.meta <scrapy.Request.meta>` key+overrides :setting:`RETRY_PRIORITY_ADJUST` for a request.+To retry requests from a spider callback, you can use the:func:`get_retry_request` function:@@ -1088,6 +1113,8 @@ Adjust retry request priority relative to original request:- a positive priority adjust means higher priority.- **a negative priority adjust (default) means lower priority.**+See also: :reqmeta:`priority_adjust`.+.. _topics-dlmw-robots:diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 7344292a2..4a6d19ce2 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -161,7 +161,7 @@ Request objectsA common use of request metadata is to define request-specificparameters for Scrapy components (extensions, middlewares, etc.). For- example, if you set ``dont_retry`` to ``True``,+ example, if you set :reqmeta:`dont_retry` to ``True``,:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` will neverretry that request, even if it fails. See :ref:`topics-request-meta`.@@ -169,15 +169,15 @@ Request objectsexample, to keep request state information relevant to your component.For example,:class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` uses the- ``retry_times`` metadata key to keep track of how many times a request- has been retried so far.+ :reqmeta:`retry_times` metadata key to keep track of how many times a+ request has been retried so far.Copying all the metadata of a previous request into a new, follow-uprequest in a spider callback is a bad practice, because requestmetadata may include metadata set by Scrapy components that is notmeant to be copied into other requests. For example, copying the- ``retry_times`` metadata key into follow-up requests can lower the- amount of retries allowed for those follow-up requests.+ :reqmeta:`retry_times` metadata key into follow-up requests can lower+ the amount of retries allowed for those follow-up requests.You should only copy all request metadata from one request to anotherif the new request is meant to replace the old request, as is often the@@ -828,6 +828,7 @@ Those are:* :reqmeta:`autothrottle_dont_adjust_delay`* :reqmeta:`bindaddress`* :reqmeta:`cookiejar`+* :reqmeta:`depth`* :reqmeta:`dont_cache`* :reqmeta:`dont_merge_cookies`* :reqmeta:`dont_obey_robotstxt`@@ -837,10 +838,12 @@ Those are:* :reqmeta:`download_latency`* :reqmeta:`download_maxsize`* :reqmeta:`download_slot`-* :reqmeta:`download_warnsize`* :reqmeta:`download_timeout`-* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)-* ``ftp_user`` (See :setting:`FTP_USER` for more info)+* :reqmeta:`download_warnsize`+* :reqmeta:`ftp_local_filename`+* :reqmeta:`ftp_passive`+* :reqmeta:`ftp_password`+* :reqmeta:`ftp_user`* :reqmeta:`give_up_log_level`* :reqmeta:`handle_httpstatus_all`* :reqmeta:`handle_httpstatus_list`@@ -848,13 +851,22 @@ Those are:* :reqmeta:`http_pass`* :reqmeta:`http_user`* :reqmeta:`is_start_request`+* :reqmeta:`link_text`* :reqmeta:`max_retry_times`+* :reqmeta:`priority_adjust`* :reqmeta:`proxy`* :reqmeta:`redirect_reasons`+* :reqmeta:`redirect_times`+* :reqmeta:`redirect_ttl`* :reqmeta:`redirect_urls`* :reqmeta:`referrer_policy`+* :reqmeta:`retry_times`+* :reqmeta:`rule`* :reqmeta:`verbatim_url`+Scrapy components also use meta keys whose name starts with an underscore, such+as ``_auth_proxy``. Those are internal, and may change or disappear at any time.+.. reqmeta:: bindaddressbindaddress@@ -921,6 +933,15 @@ download_fail_on_datalossWhether or not to fail on broken responses. See::setting:`DOWNLOAD_FAIL_ON_DATALOSS`.+.. reqmeta:: ftp_local_filename++ftp_local_filename+------------------++Path, as :class:`bytes`, of the file where to write the response body of an+``ftp://`` request. If set, :attr:`Response.body <scrapy.http.Response.body>`+holds this path instead of the file contents.+.. reqmeta:: give_up_log_levelgive_up_log_leveldiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex f7f5bf8ae..aa4208607 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1085,6 +1085,7 @@ handler (without replacement), place this in your ``settings.py``::ref:`security-local-resources`+.. reqmeta:: download_slot.. setting:: DOWNLOAD_SLOTSDOWNLOAD_SLOTS@@ -1109,6 +1110,14 @@ Allows to define concurrency/delay parameters on per slot (domain) basis:- :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`: ``concurrency``- :setting:`RANDOMIZE_DOWNLOAD_DELAY`: ``randomize_delay``+Requests are assigned to a slot based on their URL domain. To assign a request+to a specific slot instead, set the name of the slot as the ``download_slot``+:attr:`Request.meta <scrapy.Request.meta>` key. Once a request is assigned to a+slot, that key holds the name of the slot.++Since that key is kept on redirects, a redirected request stays in the slot of+the request it comes from, even when it points to a different domain.+.. setting:: DOWNLOAD_TIMEOUT@@ -1438,6 +1447,7 @@ non-default value in :ref:`per-spider settings <spider-settings>`... note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.+.. reqmeta:: ftp_passive.. setting:: FTP_PASSIVE_MODEFTP_PASSIVE_MODE@@ -1445,7 +1455,8 @@ FTP_PASSIVE_MODEDefault: ``True``-Whether or not to use passive mode when initiating FTP transfers.+Whether or not to use passive mode when initiating FTP transfers, unless there+is an ``"ftp_passive"`` key in ``Request`` meta... note::diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 219eff206..cb5608ac6 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -233,6 +233,8 @@ DepthMiddleware.. module:: scrapy.spidermiddlewares.depth:synopsis: Depth Spider Middleware+.. reqmeta:: depth+.. autoclass:: DepthMiddlewareHttpErrorMiddlewarediff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex a932dc72f..ae79ec3ee 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -491,6 +491,15 @@ CrawlSpiderdescribed below. If multiple rules match the same link, the first onewill be used, according to the order they're defined in this attribute.+ .. reqmeta:: rule++ Requests generated from :attr:`rules` carry the index of the matching rule+ within :attr:`rules` in their ``rule``+ :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs+ that key to dispatch the response to the right rule, so copying it into a+ request generated by a different rule sends the response to the wrong+ callback.+This spider also exposes an overridable method:.. method:: parse_start_url(response, **kwargs)@@ -504,6 +513,8 @@ CrawlSpiderCrawling rules~~~~~~~~~~~~~~+.. reqmeta:: link_text+.. autoclass:: Rule``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object whichdiff --git a/scrapy/http/cookies.py b/scrapy/http/cookies.pyindex 555d930e6..8008ca9d2 100644--- a/scrapy/http/cookies.py+++ b/scrapy/http/cookies.py@@ -154,14 +154,11 @@ class WrappedRequest:return urlparse_cached(self.request).schemedef is_unverifiable(self) -> bool:- """Unverifiable should indicate whether the request is unverifiable, as defined by RFC 2965.-- It defaults to False. An unverifiable request is one whose URL the user did not have the- option to approve. For example, if the request is for an image in an- HTML document, and the user had no option to approve the automatic- fetching of the image, this should be true.+ """Return ``False``, as Scrapy does not track whether the user had the+ option to approve the URL of a request, which is what makes a request+ unverifiable as defined by :rfc:`2965`."""- return cast("bool", self.request.meta.get("is_unverifiable", False))+ return False@propertydef full_url(self) -> str:diff --git a/tests/test_http_cookies.py b/tests/test_http_cookies.pyindex 5d4d5a2bd..8840172a5 100644--- a/tests/test_http_cookies.py+++ b/tests/test_http_cookies.py@@ -79,11 +79,6 @@ class TestWrappedRequest:assert not self.wrapped.is_unverifiable()assert not self.wrapped.unverifiable- def test_is_unverifiable2(self):- self.request.meta["is_unverifiable"] = True- assert self.wrapped.is_unverifiable()- assert self.wrapped.unverifiable-def test_get_origin_req_host(self):assert self.wrapped.origin_req_host == "www.example.com"