Scrapy, a fast high-level web crawling & scraping framework for Python.
[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider
Eugenio Lacuesta8a1dc26d4662
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex dd763b607..406f50fb3 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -392,11 +392,6 @@ Crawling rulesobject will contain the text of the link that produced the :class:`~scrapy.http.Request`in its ``meta`` dictionary (under the ``link_text`` key)- .. warning:: When writing crawl spider rules, avoid using ``parse`` as- callback, since the :class:`CrawlSpider` uses the ``parse`` method- itself to implement its logic. So if you override the ``parse`` method,- the crawl spider will no longer work.-``cb_kwargs`` is a dict containing the keyword arguments to be passed to thecallback function.@@ -422,6 +417,12 @@ Crawling rulesIt receives a :class:`Twisted Failure <twisted.python.failure.Failure>`instance as first parameter.++.. warning:: Because of its internal implementation, you must explicitly set+ callbacks for new requests when writing :class:`CrawlSpider`-based spiders;+ unexpected behaviour can occur otherwise.++CrawlSpider example~~~~~~~~~~~~~~~~~~~@@ -452,6 +453,11 @@ Let's now take a look at an example CrawlSpider with rules::item['name'] = response.xpath('//td[@id="item_name"]/text()').get()item['description'] = response.xpath('//td[@id="item_description"]/text()').get()item['link_text'] = response.meta['link_text']+ url = response.xpath('//td[@id="additional_data"]/@href').get()+ return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))++ def parse_additional_page(self, response, item):+ item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()return item@@ -545,6 +551,11 @@ XMLFeedSpiderthose results. It must return a list of results (Items or Requests).+.. warning:: Because of its internal implementation, you must explicitly set+ callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;+ unexpected behaviour can occur otherwise.++XMLFeedSpider example~~~~~~~~~~~~~~~~~~~~~