Scrapy, a fast high-level web crawling & scraping framework for Python.
Use autodoc for CrawlSpider and Rule (#8240)
Adrianeb744fe9633e
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex 34a1dc8e4..e6d428acb 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -768,10 +768,10 @@ much because of a programming mistake. This can be configured in theHopefully by now you have a good understanding of how to use the mechanismof following links and callbacks with Scrapy.-As yet another example spider that leverages the mechanism of following links,-check out the :class:`~scrapy.spiders.CrawlSpider` class for a generic-spider that implements a small rules engine that you can use to write your-crawlers on top of it.+If the link following of a crawl can be expressed as a set of declarative+rules, the :class:`~scrapy.spiders.CrawlSpider` class implements a small rules+engine that handles it for you. For anything else, keep following links from+your callbacks, as shown above.Also, a common pattern is to build an item with data from more than one page,using a :ref:`trick to pass additional data to the callbacksdiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex a0e7eeef2..53f95e2bb 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -465,42 +465,9 @@ with a ``TestItem`` declared in a ``myproject.items`` module:CrawlSpider------------.. class:: CrawlSpider-- This is the most commonly used spider for crawling regular websites, as it- provides a convenient mechanism for following links by defining a set of rules.- It may not be the best suited for your particular web sites or project, but- it's generic enough for several cases, so you can start from it and override it- as needed for more custom functionality, or just implement your own spider.-- Apart from the attributes inherited from Spider (that you must- specify), this class supports a new attribute:-- .. attribute:: rules-- Which is a list of one (or more) :class:`Rule` objects. Each :class:`Rule`- defines a certain behaviour for crawling the site. Rules objects are- described below. If multiple rules match the same link, the first one- will be used, according to the order they're defined in this attribute.-- .. reqmeta:: rule-- Requests generated from :attr:`rules` carry the index of the matching rule- within :attr:`rules` in their ``rule``- :attr:`Request.meta <scrapy.Request.meta>` key. :class:`CrawlSpider` needs- that key to dispatch the response to the right rule, so copying it into a- request generated by a different rule sends the response to the wrong- callback.-- This spider also exposes an overridable method:-- .. method:: parse_start_url(response, **kwargs)-- This method is called for each response produced for the URLs in- the spider's ``start_urls`` attribute. It allows to parse- the initial responses and must return either an- :ref:`item object <topics-items>`, a :class:`~scrapy.Request`- object, or an iterable containing any of them.+.. autoclass:: CrawlSpider+ :members:+ :exclude-members: from_crawlerCrawling rules~~~~~~~~~~~~~~@@ -509,65 +476,6 @@ Crawling rules.. autoclass:: Rule- ``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which- defines how links will be extracted from each crawled page. Each produced link will- be used to generate a :class:`~scrapy.Request` object, which will contain the- link's text in its ``meta`` dictionary (under the ``link_text`` key).- If omitted, a default link extractor created with no arguments will be used,- resulting in all links being extracted.-- ``callback`` is a callable or a string (in which case a method from the spider- object with that name will be used) to be called for each link extracted with- the specified link extractor. This callback receives a :class:`~scrapy.http.Response`- as its first argument and must return either a single instance or an iterable of- :ref:`item objects <topics-items>` and/or :class:`~scrapy.Request` objects- (or any subclass of them). As mentioned above, the received :class:`~scrapy.http.Response`- object will contain the text of the link that produced the :class:`~scrapy.Request`- in its ``meta`` dictionary (under the ``link_text`` key)-- ``cb_kwargs`` is a dict containing the keyword arguments to be passed to the- callback function.-- ``follow`` is a boolean which specifies if links should be followed from each- response extracted with this rule. If ``callback`` is None ``follow`` defaults- to ``True``, otherwise it defaults to ``False``.-- ``process_links`` is a callable, or a string (in which case a method from the- spider object with that name will be used) which will be called for each list- of links extracted from each response using the specified ``link_extractor``.- This is mainly used for filtering purposes.-- ``process_request`` is a callable (or a string, in which case a method from- the spider object with that name will be used) which will be called for every- :class:`~scrapy.Request` extracted by this rule. This callable should- take said request as first argument and the :class:`~scrapy.http.Response`- from which the request originated as second argument. It must return a- ``Request`` object or ``None`` (to filter out the request).-- Use ``process_request`` to set the :attr:`~scrapy.Request.priority` of- requests generated by a rule, e.g. ``process_request=lambda request,- response: request.replace(priority=10)``.-- ``errback`` is a callable or a string (in which case a method from the spider- object with that name will be used) to be called if any exception is- raised while processing a request generated by the rule.- It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`- instance as first parameter.-- ``name`` is a string that identifies the rule, to be used as a target of- other rules' ``from_rules``.-- ``from_rules`` is a string, or an iterable of strings, with the ``name``- of other rules. If set, this rule is only applied to responses reached- through one of those rules, instead of to every response.-- .. versionadded:: VERSION- The ``name`` and ``from_rules`` parameters.-- .. warning:: Because of its internal implementation, you must explicitly set- callbacks for new requests when writing :class:`CrawlSpider`-based spiders;- unexpected behaviour can occur otherwise.-CrawlSpider example~~~~~~~~~~~~~~~~~~~diff --git a/scrapy/spiders/crawl.py b/scrapy/spiders/crawl.pyindex e6e0716bc..9daad7bdc 100644--- a/scrapy/spiders/crawl.py+++ b/scrapy/spiders/crawl.py@@ -59,6 +59,66 @@ _default_link_extractor = LinkExtractor()class Rule:+ """A link-following rule for :class:`CrawlSpider`, to be used in its+ :attr:`~scrapy.spiders.CrawlSpider.rules` attribute.++ It defines which links to extract from a response, what to do with the+ responses those links produce, and whether to keep extracting links from+ them.++ .. versionadded:: VERSION+ The *name* and *from_rules* parameters.++ *callback*, *process_links*, *process_request* and *errback* may be+ specified as a string instead of a callable, in which case the spider+ method with that name is used.++ *link_extractor* is a :class:`LinkExtractor+ <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` object which defines+ how links will be extracted from each crawled page. Each produced link will+ be used to generate a :class:`~scrapy.Request` object, which will contain+ the link's text in its ``meta`` dictionary (under the ``link_text`` key).+ If omitted, a default link extractor created with no arguments will be+ used, resulting in all links being extracted.++ *callback* is called for each link extracted with *link_extractor*. It is a+ :meth:`spider callback <scrapy.Spider.parse>`. If omitted, no callback runs+ for the matching responses, which are then only used to extract further+ links, as per *follow*.++ *cb_kwargs* are the keyword arguments to pass to the callback.++ *follow* determines whether links should be followed from each response+ extracted with this rule. It defaults to ``True`` if *callback* is+ ``None``, and to ``False`` otherwise.++ *process_links* is called for each list of links extracted from each+ response using *link_extractor*. This is mainly used for filtering+ purposes.++ *process_request* is called for every :class:`~scrapy.Request` extracted by+ this rule. It should take said request as first argument and the+ :class:`~scrapy.http.Response` from which the request originated as second+ argument. It must return a :class:`~scrapy.Request` object or ``None`` (to+ filter out the request).++ Use *process_request* to set the :attr:`~scrapy.Request.priority` of+ requests generated by a rule, e.g. ``process_request=lambda request,+ response: request.replace(priority=10)``.++ *errback* is called if any exception is raised while processing a request+ generated by the rule. It receives a+ :class:`Twisted Failure <twisted.python.failure.Failure>` instance as first+ parameter.++ *name* identifies the rule, to be used as a target of the *from_rules* of+ other rules.++ *from_rules* is the *name* of another rule, or an iterable of names. If+ set, this rule is only applied to responses reached through one of those+ rules, instead of to every response.+ """+def __init__(self,link_extractor: LinkExtractor | None = None,@@ -100,7 +160,46 @@ class Rule:class CrawlSpider(Spider):+ """Spider that follows links based on a set of rules.++ You declare a set of :class:`Rule` objects in :attr:`rules`, and this+ spider extracts links from every response it gets and follows them+ accordingly.++ Use it only for crawls that fit that declarative model. As soon as you need+ to control which requests are sent, in which order, or which data they+ carry along, subclass :class:`~scrapy.Spider` instead and follow links from+ your callbacks, e.g. with :meth:`response.follow+ <scrapy.http.TextResponse.follow>`. That is usually simpler than bending+ :attr:`rules` to a use case they do not cover.++ .. warning:: Unlike in other spiders, a request without a callback is not+ handled by :meth:`Spider.parse <scrapy.Spider.parse>`. It is handled by+ :meth:`parse_start_url`, and its links are followed according to+ :attr:`rules`.++ That is the way to feed a response of your own to :attr:`rules`. If you+ want your own parsing code to run for it instead, set its callback+ explicitly; defining :meth:`~scrapy.Spider.parse` is not enough.++ Requests generated from :attr:`rules` are not affected, they always get+ a callback of their own.++ .. reqmeta:: rule++ Requests generated from :attr:`rules` carry the index of the matching rule+ within :attr:`rules` in their ``rule`` :attr:`Request.meta+ <scrapy.Request.meta>` key. :class:`CrawlSpider` needs that key to dispatch+ the response to the right rule, so copying it into a request generated by a+ different rule sends the response to the wrong callback.+ """+rules: Sequence[Rule] = ()+ """The rules that define how to crawl the site. If multiple rules match the+ same link, the first one will be used, according to the order they're+ defined in this attribute.+ """+_rules: list[Rule]_follow_links: bool@@ -127,6 +226,12 @@ class CrawlSpider(Spider):)def parse_start_url(self, response: Response, **kwargs: Any) -> Any:+ """Handle *response*, one of the responses produced for the URLs in the+ :attr:`~scrapy.Spider.start_urls` attribute of the spider.++ This is a :meth:`spider callback <scrapy.Spider.parse>`; override it to+ parse the initial responses.+ """return ()def process_results(