Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Update docs/topics/settings.rst

Deepanshu vermafbb1236fd6ee
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 4d3ae20cc..ed8f1f105 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1645,12 +1645,19 @@ Default: ``2083``
Scope: ``spidermiddlewares.urllength``
-The maximum URL length to allow for crawled URLs. You can set this to ``0``
-to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working
-with URLs longer than the default value. The default limit acts as a stopping condition in case of
-URLs of increasing length, usually caused by a loop.
-For more information about the default value
-for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
+The maximum URL length to allow for crawled URLs.
+
+This setting can act as a stopping condition in case of URLs of ever-increasing
+length, which may be caused for example by a programming error either in the
+target server or in your code. See also :setting:`REDIRECT_MAX_TIMES` and
+:setting:`DEPTH_LIMIT`.
+
+Use ``0`` to allow URLs of any length.
+
+The default value is copied from the `Microsoft Internet Explorer maximum URL
+length`_, even though this setting exists for different reasons.
+
+.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
.. setting:: USER_AGENT