Scrapy, a fast high-level web crawling & scraping framework for Python.
add link to google description of lenght-based rule
Vostretsov Nikitad39ef77e6be5
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex de5f72b80..9ce4293d2 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -1023,10 +1023,10 @@ Parsers vary in several aspects:* Support for wildcard matching-* Usage of length based rule: in particular for ``Allow`` and- ``Disallow`` directives, where the most specific rule based on the length of- the path trumps the less specific (shorter) rule-+* Usage of `length based rule <https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-group-member-lines>`_:+ in particular for ``Allow`` and ``Disallow`` directives, where the most+ specific rule based on the length of the path trumps the less specific+ (shorter) rule.. _protego-parser: