Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider

Eugenio Lacuesta8a1dc26d4662
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index dd763b607..406f50fb3 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -392,11 +392,6 @@ Crawling rules
object will contain the text of the link that produced the :class:`~scrapy.http.Request`
in its ``meta`` dictionary (under the ``link_text`` key)
- .. warning:: When writing crawl spider rules, avoid using ``parse`` as
- callback, since the :class:`CrawlSpider` uses the ``parse`` method
- itself to implement its logic. So if you override the ``parse`` method,
- the crawl spider will no longer work.
-
``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
callback function.
@@ -422,6 +417,12 @@ Crawling rules
It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
instance as first parameter.
+
+.. warning:: Because of its internal implementation, you must explicitly set
+ callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
+ unexpected behaviour can occur otherwise.
+
+
CrawlSpider example
~~~~~~~~~~~~~~~~~~~
@@ -452,6 +453,11 @@ Let's now take a look at an example CrawlSpider with rules::
item['name'] = response.xpath('//td[@id="item_name"]/text()').get()
item['description'] = response.xpath('//td[@id="item_description"]/text()').get()
item['link_text'] = response.meta['link_text']
+ url = response.xpath('//td[@id="additional_data"]/@href').get()
+ return response.follow(url, self.parse_additional_page, cb_kwargs=dict(item=item))
+
+ def parse_additional_page(self, response, item):
+ item['additional_data'] = response.xpath('//p[@id="additional_data"]/text()').get()
return item
@@ -545,6 +551,11 @@ XMLFeedSpider
those results. It must return a list of results (Items or Requests).
+.. warning:: Because of its internal implementation, you must explicitly set
+ callbacks for new requests when writing :class:`XMLFeedSpider`-based spiders;
+ unexpected behaviour can occur otherwise.
+
+
XMLFeedSpider example
~~~~~~~~~~~~~~~~~~~~~