Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: recipe for deduplicating query params in request fingerprints (#8209)
Adrian968d425d907d
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 46a2fc91f..39f8701f1 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -378,6 +378,35 @@ account:def fingerprint(self, request):return fingerprint(request, include_headers=["X-ID"])+To deduplicate repeated query string parameters, such as those some sites+add on every redirect and can otherwise cause redirect loops, build the+deduplicated URL yourself and delegate the rest to+:func:`scrapy.utils.request.fingerprint`:++.. code-block:: python++ # my_project/settings.py+ REQUEST_FINGERPRINTER_CLASS = "my_project.utils.RequestFingerprinter"++ # my_project/utils.py+ from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit+ from weakref import WeakKeyDictionary++ from scrapy.utils.request import fingerprint+++ class RequestFingerprinter:+ cache = WeakKeyDictionary()++ def fingerprint(self, request):+ if request not in self.cache:+ parts = urlsplit(request.url)+ query = urlencode(list(set(parse_qsl(parts.query))))+ deduped_url = urlunsplit(parts._replace(query=query))+ deduped_request = request.replace(url=deduped_url)+ self.cache[request] = fingerprint(deduped_request)+ return self.cache[request]+You can also write your own fingerprinting logic from scratch.However, if you do not use :func:`scrapy.utils.request.fingerprint`, make sure