Scrapy, a fast high-level web crawling & scraping framework for Python.
documentation rework
Vostretsov Nikita171fa1cd106f
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 192bfd19a..52be8ded2 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -1000,10 +1000,10 @@ RobotsTxtMiddlewareScrapy ships with support for the following robots.txt_ parsers:- * :ref:`RobotFileParser <python-robotfileparser>` (default)+ * :ref:`Protego <protego-parser>` (default)+ * :ref:`RobotFileParser <python-robotfileparser>`* :ref:`Reppy <reppy-parser>`* :ref:`Robotexclusionrulesparser <rerp-parser>`- * :ref:`Protego <protego-parser>`You can change the robots.txt_ parser with the :setting:`ROBOTSTXT_PARSER`setting. Or you can also :ref:`implement support for a new parser <support-for-new-robots-parser>`.@@ -1015,50 +1015,78 @@ If :attr:`Request.meta <scrapy.http.Request.meta>` hasthe request will be ignored by this middleware even if:setting:`ROBOTSTXT_OBEY` is enabled.+Parsers varies in several aspects:++* Language of implementation++* Supported specification++* Support for wildcard matching++* usage of length based rule: in particular for ``Allow`` and+ ``Disallow`` directives, where the most specific rule based on the length of+ the path trumps the less specific (shorter) rule+++.. _protego-parser:++Protego parser+~~~~~~~~~~~~~~++based on `Protego <https://github.com/scrapy/protego>`_:++* implemented in Python++* is compliant with `Google's Robots.txt Specification+ <https://developers.google.com/search/reference/robots_txt>`_++* supports wildcard matching++* uses the length based rule,++Scrapy uses this parser by default.+.. _python-robotfileparser:RobotFileParser~~~~~~~~~~~~~~~-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ is-Python's built-in robots.txt_ parser. The parser is fully compliant with `Martijn Koster's-1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_. It lacks-support for wildcard matching.+based on `RobotFileParser+<https://docs.python.org/3.7/library/urllib.robotparser.html>`_:-In order to use this parser, set:+* is Python's built-in robots.txt_ parser.-* :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser``+* is compliant with `Martijn Koster's 1996 draft specification+ <http://www.robotstxt.org/norobots-rfc.txt>`_.-.. _rerp-parser:--Robotexclusionrulesparser-~~~~~~~~~~~~~~~~~~~~~~~~~+* lacks support for wildcard matching.-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ is fully compliant-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,-with support for wildcard matching.+* doesn't use the length based rule,-In order to use this parser:+It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0 .-* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running- ``pip install robotexclusionrulesparser``+In order to use this parser, set:-* Set :setting:`ROBOTSTXT_PARSER` setting to- ``scrapy.robotstxt.RerpRobotParser``+* :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser``.. _reppy-parser:Reppy parser~~~~~~~~~~~~-`Reppy <https://github.com/seomoz/reppy/>`_ is a Python wrapper around `Robots Exclusion-Protocol Parser for C++ <https://github.com/seomoz/rep-cpp>`_. The parser is fully compliant-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,-with support for wildcard matching. Unlike-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ and-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_, it uses the length based-rule, in particular for ``Allow`` and ``Disallow`` directives, where the most specific-rule based on the length of the path trumps the less specific (shorter) rule.+based on `Reppy <https://github.com/seomoz/reppy/>`_:++* is a Python wrapper around `Robots Exclusion Protocol Parser for C+++ <https://github.com/seomoz/rep-cpp>`_.++* is compliant with `Martijn Koster's 1996 draft specification+ <http://www.robotstxt.org/norobots-rfc.txt>`_.++* supports wildcard matching++* uses the length based rule,++Native implementation provides better speed than Protego.In order to use this parser:@@ -1067,21 +1095,29 @@ In order to use this parser:* Set :setting:`ROBOTSTXT_PARSER` setting to``scrapy.robotstxt.ReppyRobotParser``-.. _protego-parser:+.. _rerp-parser:-Protego parser-~~~~~~~~~~~~~~+Robotexclusionrulesparser+~~~~~~~~~~~~~~~~~~~~~~~~~-`Protego <https://github.com/scrapy/protego>`_ is a pure-Python robots.txt_ parser.-The parser is fully compliant with `Google's Robots.txt Specification-<https://developers.google.com/search/reference/robots_txt>`_ hence supports wildcard-matching, and uses the length based rule similar to `Reppy <https://github.com/seomoz/reppy/>`_.-Scrapy uses this parser by default.+based on `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_:++* implemented in Python++* is compliant with `Martijn Koster's 1996 draft specification+ <http://www.robotstxt.org/norobots-rfc.txt>`_.++* supports wildcard matching++* doesn't use the length based rule,In order to use this parser:+* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running+ ``pip install robotexclusionrulesparser``+* Set :setting:`ROBOTSTXT_PARSER` setting to- ``scrapy.robotstxt.ProtegoRobotParser``+ ``scrapy.robotstxt.RerpRobotParser``.. _support-for-new-robots-parser: