Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

[wip] changing introduction to scraping with selectors

Elias Dorneles2a409d1d951d
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index 79cf502a6..f6aa6476c 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -180,235 +180,120 @@ is Scrapy's default callback method.
Extracting data
---------------
-Introduction to Selectors
-^^^^^^^^^^^^^^^^^^^^^^^^^
+The best way to learn how to extract data with Scrapy is trying selectors
+using the shell :ref:`Scrapy shell <topics-shell>`. Run::
-There are several ways to extract data from web pages. Scrapy uses a mechanism
-based on `XPath`_ or `CSS`_ expressions called :ref:`Scrapy Selectors
-<topics-selectors>`. For more information about selectors and other extraction
-mechanisms see the :ref:`Selectors documentation <topics-selectors>`.
+ scrapy crawl http://quotes.toscrape.com/page/1/
-.. _XPath: https://www.w3.org/TR/xpath
-.. _CSS: https://www.w3.org/TR/selectors
-
-Here are some examples of XPath expressions, their meanings and CSS
-equivalents:
-
-* ``/html/head/title``: selects the ``<title>`` element, inside the ``<head>``
- element of an HTML document. Using CSS, the equivalent would be: ``html >
- head > title``.
-
-* ``/html/head/title/text()``: selects the text inside the aforementioned
- ``<title>`` element. In Scrapy, you can do the same with CSS using ``html >
- head > title::text``. The ``::text`` bit isn't really CSS, but is supported
- by Scrapy for extracting purposes.
-
-* ``//td``: selects all the ``<td>`` elements from the whole document.
- The equivalent CSS selector: ``td``.
-
-* ``//div[@id="mine"]``: selects all ``div`` elements which contain an
- attribute ``id="mine"``. The equivalent CSS selector would be: ``div#mine``.
-
-These are just a couple of simple examples of what you can do with XPath and
-CSS. XPath expressions are very powerful, they're the foundation of Scrapy
-selectors. In fact, CSS selectors are converted to XPath expressions
-under-the-hood. To learn more about XPath, we recommend `this tutorial to learn
-XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
-tutorial to learn "how to think in XPath"
-<http://plasmasturm.org/log/xpath101/>`_.
-
-.. note:: **CSS vs XPath:** you can go a long way extracting data from web pages
- using only CSS selectors. However, XPath offers more power because besides
- navigating the structure, it can also look at the content: you're
- able to select things like: *the link that contains the text 'Next Page'*.
- Because of this, we encourage you to learn about XPath even if you
- already know how to construct CSS selectors.
-
-For working with CSS and XPath expressions, Scrapy provides the
-:class:`~scrapy.selector.Selector` class and convenient shortcuts to avoid
-instantiating selectors yourself every time you need to select something from a
-response.
-
-You can see selectors as objects that represent nodes in the document
-structure. So, the first instantiated selectors are associated with the root
-node, or the entire document.
-
-Selectors have four basic methods (click on the method to see the complete API
-documentation):
-
-* :meth:`~scrapy.selector.Selector.xpath`: returns a list of selectors, each of
- which represents the nodes selected by the xpath expression given as
- argument.
-
-* :meth:`~scrapy.selector.Selector.css`: returns a list of selectors, each of
- which represents the nodes selected by the CSS expression given as argument.
-
-* :meth:`~scrapy.selector.Selector.extract`: returns a unicode string with the
- selected data.
-
-* :meth:`~scrapy.selector.Selector.re`: returns a list of unicode strings
- extracted by applying the regular expression given as argument.
-
-
-Trying Selectors in the Shell
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy
-shell <topics-shell>`, which also requires `IPython <http://ipython.org/>`_ (an extended Python console)
-installed on your system.
-
-To start a shell, you must go to the project's top level directory and run::
-
- scrapy shell "http://quotes.toscrape.com"
-
-.. note::
-
- Remember to always enclose urls in quotes when running Scrapy shell from
- command-line, otherwise urls containing arguments (ie. ``&`` character)
- will not work.
-
-This is what the shell looks like::
+You will see something like::
[ ... Scrapy log here ... ]
-
- 2016-09-01 18:14:39 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com> (referer: None)
+ 2016-09-19 12:09:27 [scrapy] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects:
- [s] crawler <scrapy.crawler.Crawler object at 0x109001c90>
+ [s] crawler <scrapy.crawler.Crawler object at 0x7fa91d888c90>
[s] item {}
- [s] request <GET http://quotes.toscrape.com>
- [s] response <200 http://quotes.toscrape.com>
- [s] settings <scrapy.settings.Settings object at 0x109001610>
- [s] spider <DefaultSpider 'default' at 0x1092808d0>
+ [s] request <GET http://quotes.toscrape.com/page/1/>
+ [s] response <200 http://quotes.toscrape.com/page/1/>
+ [s] settings <scrapy.settings.Settings object at 0x7fa91d888c10>
+ [s] spider <DefaultSpider 'default' at 0x7fa91c8af990>
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
-
- >>>
-
-After the shell loads, you will have the response fetched in a local
-``response`` variable, so if you type ``response.body`` you will see the body
-of the response, or you can type ``response.headers`` to see its headers.
-
-More importantly ``response`` has a ``selector`` attribute which is an instance of
-:class:`~scrapy.selector.Selector` class, instantiated with this particular ``response``.
-You can run queries on ``response`` by calling ``response.selector.xpath()`` or
-``response.selector.css()``. There are also some convenience shortcuts like ``response.xpath()``
-or ``response.css()`` which map directly to ``response.selector.xpath()`` and
-``response.selector.css()``.
-
-
-So let's try it::
-
- In [1]: response.xpath('//title')
- Out[1]: [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]
-
- In [2]: response.xpath('//title').extract()
- Out[2]: [u'<title>Quotes to Scrape</title>']
-
- In [3]: response.xpath('//title/text()')
- Out[3]: [<Selector xpath='//title/text()' data=u'Quotes to Scrape'>]
-
- In [4]: response.xpath('//title/text()').extract()
- Out[4]: [u'Quotes to Scrape']
-
- In [11]: response.xpath('//title/text()').re('(\w+)')
- Out[11]: [u'Quotes', u'to', u'Scrape']
-
-Extracting the data
-^^^^^^^^^^^^^^^^^^^
-
-Now, let's try to extract some real information from those pages.
-
-You could type ``response.body`` in the console, and inspect the source code to
-figure out the XPaths you need to use. However, inspecting the raw HTML code
-there could become a very tedious task. To make it easier, you can
-use Firefox Developer Tools or some Firefox extensions like Firebug. For more
-information see :ref:`topics-firebug` and :ref:`topics-firefox`.
+ >>>
-After inspecting the page source, you'll find that every quote in the website
-is inside a separate ``<div class="quote">`` element, such as::
+Using the shell, you can try selecting elements using `CSS`_ with the response
+object::
- <div class="quote">
- <span class="text">“We accept the love we think we deserve.”</span>
- <span>by <small class="author">Stephen Chbosky</small></span>
- <div class="tags">
- Tags:
- <meta class="keywords">
- <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
- <a class="tag" href="/tag/love/page/1/">love</a>
- </div>
- </div>
+ >>> response.css('title')
+ [<Selector xpath=u'descendant-or-self::title' data=u'<title>Quotes to Scrape</title>'>]
+The result of running ``response.css('title')`` is a list-like object called
+:class:`~scrapy.selector.SelectorList`, which represents a list of
+:class:`~scrapy.selector.Selector` objects that wrap around XML/HTML elements
+and allow you to run further queries to fine-grain the selection or extract the
+data.
-So we can select each ``<div class="quote">`` element belonging to the site's
-list with this code::
+To extract the text from the title above, you can do::
- response.xpath('//div[@class="quote"]')
+ >>> response.css('title::text').extract()
+ [u'Quotes to Scrape']
-From the quote elements, we can select the texts with::
+There are two things to note here: one is that we've added ``::text`` to the
+CSS query, to mean that we want to select the text from inside the title element.
- response.xpath('//div[@class="quote"]/span[@class="text"]/text()').extract()
+The other is that the result of calling ``.extract()`` is a list, because we're
+dealing with an instance :class:`~scrapy.selector.SelectorList`. When you know
+you just want the first result, as in this case, you can do::
-The authors::
+ >>> response.css('title::text').extract_first()
+ u'Quotes to Scrape'
- response.xpath('//div[@class="quote"]/span/small/text()').extract()
+As an alternative, you could've written::
-As we've said before, each ``.xpath()`` call returns a list of selectors, so we can
-concatenate further ``.xpath()`` calls to dig deeper into a node. We are going to use
-that property here, so::
+ >>> response.css('title::text')[0].extract()
+ u'Quotes to Scrape'
- for quote in response.xpath('//div[@class="quote"]'):
- text = quote.xpath('span[@class="text"]/text()').extract_first()
- author = quote.xpath('span/small/text()').extract_first()
- print({'text': text, 'author': author})
+However, using ``.extract_first()`` avoids an ``IndexError`` and returns
+``None`` when it doesn't find any element matching the selection.
-In the above snippet we've decided to use the method ``.extract_first()``
-instead of ``.extract()``, to extract the content from the first element from a
-selector list returned by ``.xpath()``.
+There's a lesson here: for most scraping code, you want it to be resilient to
+errors due to things not being found on a page, so that even if some parts fail
+to be scraped, you can at least get **some** data.
-.. note::
+Besides the :meth:`~scrapy.selector.Selector.extract` and
+:meth:`~scrapy.selector.SelectorList.extract_first` methods, you can also use
+the :meth:`~scrapy.selector.Selector.re` method to extract using a regular
+expression::
- For a more detailed description of using nested selectors, see
- :ref:`topics-selectors-nesting-selectors` and
- :ref:`topics-selectors-relative-xpaths` in the :ref:`topics-selectors`
- documentation
+ >>> response.css('title::text').re('Quotes.*')
+ [u'Quotes to Scrape']
+ >>> response.css('title::text').re('Q\w+')
+ [u'Quotes']
+ >>> response.css('title::text').re('(\w+) to (\w+)')
+ [u'Quotes', u'Scrape']
-Knowing to use selectors, extracting data from a page is just a matter of
-yield the Python dictionaries from the callback method instead of printing
-them.
+In order to find the proper CSS selectors to use, you might find useful opening
+the response page from the shell in your web browser using ``view(response)``.
+You can use your browser developer tools or extensions like Firebug. For more
+information see :ref:`topics-firebug` and :ref:`topics-firefox`.
-Let's add the necessary code to our spider::
- import scrapy
+XPath: a brief intro
+^^^^^^^^^^^^^^^^^^^^
+Besides CSS, Scrapy selectors also support using `XPath`_ expressions::
- class QuotesSpider(scrapy.Spider):
- name = "quotes"
- start_urls = [
- 'http://quotes.toscrape.com/page/1/',
- 'http://quotes.toscrape.com/page/2/',
- ]
+ >>> response.xpath('//title')
+ [<Selector xpath='//title' data=u'<title>Quotes to Scrape</title>'>]
+ >>> response.xpath('//title/text()').extract_first()
+ u'Quotes to Scrape'
- def parse(self, response):
- for quote in response.xpath('//div[@class="quote"]'):
- yield {
- 'text': quote.xpath('span[@class="text"]/text()').extract_first(),
- 'author': quote.xpath('span/small/text()').extract_first(),
- }
+XPath expressions are very powerful, and are the foundation of Scrapy
+Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
+can see that if you read closely the text representation of the selector
+objects in the shell.
+
+While perhaps not as popular as CSS selectors, XPath expressions offer more
+power because besides navigating the structure, it can also look at the
+content. Using XPath, you're able to select things like: **select the link
+that contains the text "Next Page"**. This makes XPath very fitting to the task
+of scraping, and we encourage you to learn XPath even if you already know how to
+construct CSS selectors, it will make scraping much easier.
+
+We won't cover much of XPath here. To learn more about XPath, we recommend `this tutorial to learn
+XPath through examples <http://zvon.org/comp/r/tut-XPath_1.html>`_, and `this
+tutorial to learn "how to think in XPath"
+<http://plasmasturm.org/log/xpath101/>`_.
-Run::
- scrapy crawl quotes
+Extraction wrap-up
+^^^^^^^^^^^^^^^^^^
-Now crawling quotes.toscrape.com will show dictionary objects::
+Now that you know a bit about selection and extraction, let's complete our
+spider by writing the code to extract the quotes from the webpage.
- 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
- {'author': 'Oscar Wilde',
- 'text': '“We are all in the gutter, but some of us are looking at the stars.”'}
- 2016-09-02 16:35:20 [scrapy] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/2/>
- {'author': 'Mark Twain',
- 'text': '“The man who does not read has no advantage over the man who cannot read.”'}
+TODO: show how to extract quotes and integrate spider code here.
Following links