Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Warn against using keep_fragments for JS-rendered fragment content (#8033)

Adriand2ec88fb4024
diff --git a/scrapy/utils/request.py b/scrapy/utils/request.py
index b2fd9a834..b8eb633c2 100644
--- a/scrapy/utils/request.py
+++ b/scrapy/utils/request.py
@@ -69,6 +69,10 @@ def fingerprint(
so they are also ignored by default when calculating the fingerprint.
If you want to include them, set the keep_fragments argument to True
(for instance when handling requests with a headless browser).
+
+ keep_fragments is not a substitute for rendering JavaScript to reach
+ content loaded based on the URL fragment; see
+ :ref:`topics-dynamic-content` for that instead.
"""
processed_include_headers: tuple[bytes, ...] | None = None
if include_headers: