Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Use autodoc for CrawlSpider and Rule (#8240)

Adrianeb744fe9633e
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 34a1dc8e4..e6d428acb 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -768,10 +768,10 @@ much because of a programming mistake. This can be configured in the
Hopefully by now you have a good understanding of how to use the mechanism
of following links and callbacks with Scrapy.
-As yet another example spider that leverages the mechanism of following links,
-check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic
-spider that implements a small rules engine that you can use to write your
-crawlers on top of it.
+If the link following of a crawl can be expressed as a set of declarative
+rules, the :class:`~scrapy.spiders.CrawlSpider` class implements a small rules
+engine that handles it for you. For anything else, keep following links from
+your callbacks, as shown above.
Also, a common pattern is to build an item with data from more than one page,
using a :ref:`trick to pass additional data to the callbacks
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index a0e7eeef2..53f95e2bb 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -465,42 +465,9 @@ with a ``TestItem`` declared in a ``myproject.items`` module:
CrawlSpider
-----------
-.. class:: CrawlSpider
-
- This is the most commonly used spider for crawling regular websites, as it
- provides a convenient mechanism for following links by defining a set of rules.
- It may not be the best suited for your particular web sites or project, but
- it's generic enough for several cases, so you can start from it and override it
- as needed for more custom functionality, or just implement your own spider.
-
- Apart from the attributes inherited from Spider (that you must
- specify), this class supports a new attribute:
-
- .. attribute:: rules
-
- Which is a list of one (or more) :class:`Rule` objects. Each :class:`Rule`
- defines a certain behaviour for crawling the site. Rules objects are
- described below. If multiple rules match the same link, the first one
- will be used, according to the order they're defined in this attribute.
-
- .. reqmeta:: rule
-
- Requests generated from :attr:`rules` carry the index of the matching rule
- within :attr:`rules` in their ``rule``
- :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs
- that key to dispatch the response to the right rule, so copying it into a
- request generated by a different rule sends the response to the wrong
- callback.
-
- This spider also exposes an overridable method:
-
- .. method:: parse_start_url(response, **kwargs)
-
- This method is called for each response produced for the URLs in
- the spider's ``start_urls`` attribute. It allows to parse
- the initial responses and must return either an
- :ref:`item object <topics-items>`, a :class:`~scrapy.Request`
- object, or an iterable containing any of them.
+.. autoclass:: CrawlSpider
+ :members:
+ :exclude-members: from_crawler
Crawling rules
~~~~~~~~~~~~~~
@@ -509,65 +476,6 @@ Crawling rules
.. autoclass:: Rule
- ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
- defines how links will be extracted from each crawled page. Each produced link will
- be used to generate a :class:`~scrapy.Request` object, which will contain the
- link's text in its ``meta`` dictionary (under the ``link_text`` key).
- If omitted, a default link extractor created with no arguments will be used,
- resulting in all links being extracted.
-
- ``callback`` is a callable or a string (in which case a method from the spider
- object with that name will be used) to be called for each link extracted with
- the specified link extractor. This callback receives a :class:`~scrapy.http.Response`
- as its first argument and must return either a single instance or an iterable of
- :ref:`item objects <topics-items>` and/or :class:`~scrapy.Request` objects
- (or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`
- object will contain the text of the link that produced the :class:`~scrapy.Request`
- in its ``meta`` dictionary (under the ``link_text`` key)
-
- ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
- callback function.
-
- ``follow`` is a boolean which specifies if links should be followed from each
- response extracted with this rule. If ``callback`` is None ``follow`` defaults
- to ``True``, otherwise it defaults to ``False``.
-
- ``process_links`` is a callable, or a string (in which case a method from the
- spider object with that name will be used) which will be called for each list
- of links extracted from each response using the specified ``link_extractor``.
- This is mainly used for filtering purposes.
-
- ``process_request`` is a callable (or a string, in which case a method from
- the spider object with that name will be used) which will be called for every
- :class:`~scrapy.Request` extracted by this rule. This callable should
- take said request as first argument and the :class:`~scrapy.http.Response`
- from which the request originated as second argument. It must return a
- ``Request`` object or ``None`` (to filter out the request).
-
- Use ``process_request`` to set the :attr:`~scrapy.Request.priority` of
- requests generated by a rule, e.g. ``process_request=lambda request,
- response: request.replace(priority=10)``.
-
- ``errback`` is a callable or a string (in which case a method from the spider
- object with that name will be used) to be called if any exception is
- raised while processing a request generated by the rule.
- It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
- instance as first parameter.
-
- ``name`` is a string that identifies the rule, to be used as a target of
- other rules' ``from_rules``.
-
- ``from_rules`` is a string, or an iterable of strings, with the ``name``
- of other rules. If set, this rule is only applied to responses reached
- through one of those rules, instead of to every response.
-
- .. versionadded:: VERSION
- The ``name`` and ``from_rules`` parameters.
-
- .. warning:: Because of its internal implementation, you must explicitly set
- callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
- unexpected behaviour can occur otherwise.
-
CrawlSpider example
~~~~~~~~~~~~~~~~~~~
diff --git a/scrapy/spiders/crawl.py b/scrapy/spiders/crawl.py
index e6e0716bc..9daad7bdc 100644
--- a/scrapy/spiders/crawl.py
+++ b/scrapy/spiders/crawl.py
@@ -59,6 +59,66 @@ _default_link_extractor = LinkExtractor()
class Rule:
+ """A link-following rule for :class:`CrawlSpider`, to be used in its
+ :attr:`~scrapy.spiders.CrawlSpider.rules` attribute.
+
+ It defines which links to extract from a response, what to do with the
+ responses those links produce, and whether to keep extracting links from
+ them.
+
+ .. versionadded:: VERSION
+ The *name* and *from_rules* parameters.
+
+ *callback*, *process_links*, *process_request* and *errback* may be
+ specified as a string instead of a callable, in which case the spider
+ method with that name is used.
+
+ *link_extractor* is a :class:`LinkExtractor
+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` object which defines
+ how links will be extracted from each crawled page. Each produced link will
+ be used to generate a :class:`~scrapy.Request` object, which will contain
+ the link's text in its ``meta`` dictionary (under the ``link_text`` key).
+ If omitted, a default link extractor created with no arguments will be
+ used, resulting in all links being extracted.
+
+ *callback* is called for each link extracted with *link_extractor*. It is a
+ :meth:`spider callback <scrapy.Spider.parse>`. If omitted, no callback runs
+ for the matching responses, which are then only used to extract further
+ links, as per *follow*.
+
+ *cb_kwargs* are the keyword arguments to pass to the callback.
+
+ *follow* determines whether links should be followed from each response
+ extracted with this rule. It defaults to ``True`` if *callback* is
+ ``None``, and to ``False`` otherwise.
+
+ *process_links* is called for each list of links extracted from each
+ response using *link_extractor*. This is mainly used for filtering
+ purposes.
+
+ *process_request* is called for every :class:`~scrapy.Request` extracted by
+ this rule. It should take said request as first argument and the
+ :class:`~scrapy.http.Response` from which the request originated as second
+ argument. It must return a :class:`~scrapy.Request` object or ``None`` (to
+ filter out the request).
+
+ Use *process_request* to set the :attr:`~scrapy.Request.priority` of
+ requests generated by a rule, e.g. ``process_request=lambda request,
+ response: request.replace(priority=10)``.
+
+ *errback* is called if any exception is raised while processing a request
+ generated by the rule. It receives a
+ :class:`Twisted Failure <twisted.python.failure.Failure>` instance as first
+ parameter.
+
+ *name* identifies the rule, to be used as a target of the *from_rules* of
+ other rules.
+
+ *from_rules* is the *name* of another rule, or an iterable of names. If
+ set, this rule is only applied to responses reached through one of those
+ rules, instead of to every response.
+ """
+
def __init__(
self,
link_extractor: LinkExtractor | None = None,
@@ -100,7 +160,46 @@ class Rule:
class CrawlSpider(Spider):
+ """Spider that follows links based on a set of rules.
+
+ You declare a set of :class:`Rule` objects in :attr:`rules`, and this
+ spider extracts links from every response it gets and follows them
+ accordingly.
+
+ Use it only for crawls that fit that declarative model. As soon as you need
+ to control which requests are sent, in which order, or which data they
+ carry along, subclass :class:`~scrapy.Spider` instead and follow links from
+ your callbacks, e.g. with :meth:`response.follow
+ <scrapy.http.TextResponse.follow>`. That is usually simpler than bending
+ :attr:`rules` to a use case they do not cover.
+
+ .. warning:: Unlike in other spiders, a request without a callback is not
+ handled by :meth:`Spider.parse <scrapy.Spider.parse>`. It is handled by
+ :meth:`parse_start_url`, and its links are followed according to
+ :attr:`rules`.
+
+ That is the way to feed a response of your own to :attr:`rules`. If you
+ want your own parsing code to run for it instead, set its callback
+ explicitly; defining :meth:`~scrapy.Spider.parse` is not enough.
+
+ Requests generated from :attr:`rules` are not affected, they always get
+ a callback of their own.
+
+ .. reqmeta:: rule
+
+ Requests generated from :attr:`rules` carry the index of the matching rule
+ within :attr:`rules` in their ``rule`` :attr:`Request.meta
+ <scrapy.Request.meta>` key. :class:`CrawlSpider` needs that key to dispatch
+ the response to the right rule, so copying it into a request generated by a
+ different rule sends the response to the wrong callback.
+ """
+
rules: Sequence[Rule] = ()
+ """The rules that define how to crawl the site. If multiple rules match the
+ same link, the first one will be used, according to the order they're
+ defined in this attribute.
+ """
+
_rules: list[Rule]
_follow_links: bool
@@ -127,6 +226,12 @@ class CrawlSpider(Spider):
)
def parse_start_url(self, response: Response, **kwargs: Any) -> Any:
+ """Handle *response*, one of the responses produced for the URLs in the
+ :attr:`~scrapy.Spider.start_urls` attribute of the spider.
+
+ This is a :meth:`spider callback <scrapy.Spider.parse>`; override it to
+ parse the initial responses.
+ """
return ()
def process_results(