Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document update URLLENGTH_LIMIT

Deepanshu74f146bbe0c4
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 2ab2020fa..4d3ae20cc 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1645,8 +1645,12 @@ Default: ``2083``
Scope: ``spidermiddlewares.urllength``
-The maximum URL length to allow for crawled URLs. For more information about
-the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
+The maximum URL length to allow for crawled URLs. You can set this to ``0``
+to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working
+with URLs longer than the default value. The default limit acts as a stopping condition in case of
+URLs of increasing length, usually caused by a loop.
+For more information about the default value
+for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
.. setting:: USER_AGENT
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 73bedf655..a0a7b1fb6 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -440,4 +440,5 @@ UrlLengthMiddleware
settings (see the settings documentation for more info):
* :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
+ If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled.