Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

documentation rework

Vostretsov Nikita171fa1cd106f
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 192bfd19a..52be8ded2 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -1000,10 +1000,10 @@ RobotsTxtMiddleware
Scrapy ships with support for the following robots.txt_ parsers:
- * :ref:`RobotFileParser <python-robotfileparser>` (default)
+ * :ref:`Protego <protego-parser>` (default)
+ * :ref:`RobotFileParser <python-robotfileparser>`
* :ref:`Reppy <reppy-parser>`
* :ref:`Robotexclusionrulesparser <rerp-parser>`
- * :ref:`Protego <protego-parser>`
You can change the robots.txt_ parser with the :setting:`ROBOTSTXT_PARSER`
setting. Or you can also :ref:`implement support for a new parser <support-for-new-robots-parser>`.
@@ -1015,50 +1015,78 @@ If :attr:`Request.meta <scrapy.http.Request.meta>` has
the request will be ignored by this middleware even if
:setting:`ROBOTSTXT_OBEY` is enabled.
+Parsers varies in several aspects:
+
+* Language of implementation
+
+* Supported specification
+
+* Support for wildcard matching
+
+* usage of length based rule: in particular for ``Allow`` and
+ ``Disallow`` directives, where the most specific rule based on the length of
+ the path trumps the less specific (shorter) rule
+
+
+.. _protego-parser:
+
+Protego parser
+~~~~~~~~~~~~~~
+
+based on `Protego <https://github.com/scrapy/protego>`_:
+
+* implemented in Python
+
+* is compliant with `Google's Robots.txt Specification
+ <https://developers.google.com/search/reference/robots_txt>`_
+
+* supports wildcard matching
+
+* uses the length based rule,
+
+Scrapy uses this parser by default.
+
.. _python-robotfileparser:
RobotFileParser
~~~~~~~~~~~~~~~
-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ is
-Python's built-in robots.txt_ parser. The parser is fully compliant with `Martijn Koster's
-1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_. It lacks
-support for wildcard matching.
+based on `RobotFileParser
+<https://docs.python.org/3.7/library/urllib.robotparser.html>`_:
-In order to use this parser, set:
+* is Python's built-in robots.txt_ parser.
-* :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser``
+* is compliant with `Martijn Koster's 1996 draft specification
+ <http://www.robotstxt.org/norobots-rfc.txt>`_.
-.. _rerp-parser:
-
-Robotexclusionrulesparser
-~~~~~~~~~~~~~~~~~~~~~~~~~
+* lacks support for wildcard matching.
-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ is fully compliant
-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,
-with support for wildcard matching.
+* doesn't use the length based rule,
-In order to use this parser:
+It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0 .
-* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running
- ``pip install robotexclusionrulesparser``
+In order to use this parser, set:
-* Set :setting:`ROBOTSTXT_PARSER` setting to
- ``scrapy.robotstxt.RerpRobotParser``
+* :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser``
.. _reppy-parser:
Reppy parser
~~~~~~~~~~~~
-`Reppy <https://github.com/seomoz/reppy/>`_ is a Python wrapper around `Robots Exclusion
-Protocol Parser for C++ <https://github.com/seomoz/rep-cpp>`_. The parser is fully compliant
-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,
-with support for wildcard matching. Unlike
-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ and
-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_, it uses the length based
-rule, in particular for ``Allow`` and ``Disallow`` directives, where the most specific
-rule based on the length of the path trumps the less specific (shorter) rule.
+based on `Reppy <https://github.com/seomoz/reppy/>`_:
+
+* is a Python wrapper around `Robots Exclusion Protocol Parser for C++
+ <https://github.com/seomoz/rep-cpp>`_.
+
+* is compliant with `Martijn Koster's 1996 draft specification
+ <http://www.robotstxt.org/norobots-rfc.txt>`_.
+
+* supports wildcard matching
+
+* uses the length based rule,
+
+Native implementation provides better speed than Protego.
In order to use this parser:
@@ -1067,21 +1095,29 @@ In order to use this parser:
* Set :setting:`ROBOTSTXT_PARSER` setting to
``scrapy.robotstxt.ReppyRobotParser``
-.. _protego-parser:
+.. _rerp-parser:
-Protego parser
-~~~~~~~~~~~~~~
+Robotexclusionrulesparser
+~~~~~~~~~~~~~~~~~~~~~~~~~
-`Protego <https://github.com/scrapy/protego>`_ is a pure-Python robots.txt_ parser.
-The parser is fully compliant with `Google's Robots.txt Specification
-<https://developers.google.com/search/reference/robots_txt>`_ hence supports wildcard
-matching, and uses the length based rule similar to `Reppy <https://github.com/seomoz/reppy/>`_.
-Scrapy uses this parser by default.
+based on `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_:
+
+* implemented in Python
+
+* is compliant with `Martijn Koster's 1996 draft specification
+ <http://www.robotstxt.org/norobots-rfc.txt>`_.
+
+* supports wildcard matching
+
+* doesn't use the length based rule,
In order to use this parser:
+* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running
+ ``pip install robotexclusionrulesparser``
+
* Set :setting:`ROBOTSTXT_PARSER` setting to
- ``scrapy.robotstxt.ProtegoRobotParser``
+ ``scrapy.robotstxt.RerpRobotParser``
.. _support-for-new-robots-parser: