Scrapy, a fast high-level web crawling & scraping framework for Python.
Document update URLLENGTH_LIMIT
Deepanshu74f146bbe0c4
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 2ab2020fa..4d3ae20cc 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1645,8 +1645,12 @@ Default: ``2083``Scope: ``spidermiddlewares.urllength``-The maximum URL length to allow for crawled URLs. For more information about-the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246+The maximum URL length to allow for crawled URLs. You can set this to ``0``+to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working+with URLs longer than the default value. The default limit acts as a stopping condition in case of+URLs of increasing length, usually caused by a loop.+For more information about the default value+for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246.. setting:: USER_AGENTdiff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 73bedf655..a0a7b1fb6 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -440,4 +440,5 @@ UrlLengthMiddlewaresettings (see the settings documentation for more info):* :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.+ If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled.