Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction

Mikhail Korobov7fdfdb7fa244
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index cf1b0ba22..2db982b37 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -25,17 +25,14 @@ either by `XPath`_ or `CSS`_ expressions.
used with HTML. `CSS`_ is a language for applying styles to HTML documents. It
defines selectors to associate those styles with specific HTML elements.
-Scrapy selectors are powered by `parsel`_ library, which uses `lxml`_ library
-under the hood. It means Scrapy selectors are very similar in speed and
-parsing accuracy to lxml.
-
-This page explains how selectors work and describes their API which is very
-small and simple, unlike the `lxml`_ API which is much bigger because the
-`lxml`_ library can be used for many other tasks, besides selecting markup
-documents.
+.. note::
+ Scrapy Selectors is a thin wrapper around `parsel`_ library; the purpose of
+ this wrapper is to provide better integration with Scrapy Response objects.
-For a complete reference of the selectors API see
-:ref:`Selector reference <topics-selectors-ref>`
+ `parsel`_ is a stand-alone web scraping library which can be used without
+ Scrapy. It uses `lxml`_ library under the hood, and implements an
+ easy API on top of lxml API. It means Scrapy selectors are very similar
+ in speed and parsing accuracy to lxml.
.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/
.. _lxml: http://lxml.de/
@@ -73,9 +70,8 @@ Constructing from response::
>>> Selector(response=response).xpath('//span/text()').get()
'good'
-For convenience, response objects expose a selector on `.selector` attribute,
-it's totally OK to use this shortcut when possible. By using it you can
-ensure the response body is parsed only once::
+For convenience, response objects expose a selector on `.selector` attribute.
+By using it you can ensure the response body is parsed only once::
>>> response.selector.xpath('//span/text()').get()
'good'
@@ -88,8 +84,10 @@ more shortcuts: ``response.xpath()`` and ``response.css()``::
>>> response.css('span::text').get()
'good'
-Usually there is no need to construct Scrapy selectors manually because of
-these shortcuts.
+Usually there is no need to construct Scrapy selectors manually:
+``response`` object is available in Spider callbacks, so in most cases
+it is more convenient to use ``response.css()`` and ``response.xpath()``
+shortcuts.
Using selectors
---------------
@@ -392,6 +390,71 @@ Use it to extract just the first matching string::
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)')
'My image 1'
+.. _old-extraction-api:
+
+extract() and extract_first()
+-----------------------------
+
+If you're a long-time Scrapy user, you're probably familiar
+with ``.extract()`` and ``.extract_first()`` selector methods. Many blog posts
+and tutorials are using them as well. These methods are still supported
+by Scrapy, there are **no plans** to deprecate them.
+
+However, Scrapy usage docs are now written using ``.get()`` and
+``.getall()`` methods. We feel that these new methods result in a more concise
+and readable code.
+
+The following examples show how these methods map to each other.
+
+1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``::
+
+ >>> response.css('a::attr(href)').get()
+ 'image1.html'
+ >>> response.css('a::attr(href)').extract_first()
+ 'image1.html'
+
+2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``::
+
+ >>> response.css('a::attr(href)').getall()
+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
+ >>> response.css('a::attr(href)').extract()
+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
+
+2. ``Selector.get()`` is the same as ``Selector.extract()``::
+
+ >>> response.css('a::attr(href)')[0].get()
+ 'image1.html'
+ >>> response.css('a::attr(href)')[0].extract()
+ 'image1.html'
+
+4. For consistency, there is also ``Selector.getall()``, which returns a list::
+
+ >>> response.css('a::attr(href)')[0].getall()
+ ['image1.html']
+
+So, the main difference is that output of ``.get()`` and ``.getall()`` methods
+is more predictable: ``.get()`` always returns a single result, ``.getall()``
+always returns a list of all extracted results. With ``.extract()`` method
+it was not always obvious if a result is a list or not; to get a single
+result either ``.extract()`` or ``.extract_first()`` should be called.
+
+
+.. _topics-selectors-xpaths:
+
+Working with XPaths
+===================
+
+Here are some tips which may help you to use XPath with Scrapy selectors
+effectively. If you are not much familiar with XPath yet,
+you may want to take a look first at this `XPath tutorial`_.
+
+.. note::
+ Some of the tips are based on `this post from ScrapingHub's blog`_.
+
+.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html
+.. _`this post from ScrapingHub's blog`: https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/
+
+
.. _topics-selectors-relative-xpaths:
Working with relative XPaths
@@ -428,6 +491,114 @@ XPath specification.
.. _Location Paths: https://www.w3.org/TR/xpath#location-paths
+When querying by class, consider using CSS
+------------------------------------------
+
+Because an element can contain multiple CSS classes, the XPath way to select elements
+by class is the rather verbose::
+
+ *[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')]
+
+If you use ``@class='someclass'`` you may end up missing elements that have
+other classes, and if you just use ``contains(@class, 'someclass')`` to make up
+for that you may end up with more elements that you want, if they have a different
+class name that shares the string ``someclass``.
+
+As it turns out, Scrapy selectors allow you to chain selectors, so most of the time
+you can just select by class using CSS and then switch to XPath when needed::
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')
+ >>> sel.css('.shout').xpath('./time/@datetime').getall()
+ ['2014-07-23 19:00']
+
+This is cleaner than using the verbose XPath trick shown above. Just remember
+to use the ``.`` in the XPath expressions that will follow.
+
+Beware of the difference between //node[1] and (//node)[1]
+----------------------------------------------------------
+
+``//node[1]`` selects all the nodes occurring first under their respective parents.
+
+``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them.
+
+Example::
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(text="""
+ ....: <ul class="list">
+ ....: <li>1</li>
+ ....: <li>2</li>
+ ....: <li>3</li>
+ ....: </ul>
+ ....: <ul class="list">
+ ....: <li>4</li>
+ ....: <li>5</li>
+ ....: <li>6</li>
+ ....: </ul>""")
+ >>> xp = lambda x: sel.xpath(x).getall()
+
+This gets all first ``<li>`` elements under whatever it is its parent::
+
+ >>> xp("//li[1]")
+ ['<li>1</li>', '<li>4</li>']
+
+And this gets the first ``<li>`` element in the whole document::
+
+ >>> xp("(//li)[1]")
+ ['<li>1</li>']
+
+This gets all first ``<li>`` elements under an ``<ul>`` parent::
+
+ >>> xp("//ul/li[1]")
+ ['<li>1</li>', '<li>4</li>']
+
+And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document::
+
+ >>> xp("(//ul/li)[1]")
+ ['<li>1</li>']
+
+Using text nodes in a condition
+-------------------------------
+
+When you need to use the text content as argument to an `XPath string function`_,
+avoid using ``.//text()`` and use just ``.`` instead.
+
+This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*.
+And when a node-set is converted to a string, which happens when it is passed as argument to
+a string function like ``contains()`` or ``starts-with()``, it results in the text for the first element only.
+
+Example::
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
+
+Converting a *node-set* to string::
+
+ >>> sel.xpath('//a//text()').getall() # take a peek at the node-set
+ ['Click here to go to the ', 'Next Page']
+ >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
+ ['Click here to go to the ']
+
+A *node* converted to a string, however, puts together the text of itself plus of all its descendants::
+
+ >>> sel.xpath("//a[1]").getall() # select the first node
+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
+ >>> sel.xpath("string(//a[1])").getall() # convert it to string
+ ['Click here to go to the Next Page']
+
+So, using the ``.//text()`` node-set won't select anything in this case::
+
+ >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
+ []
+
+But using the ``.`` to mean the node, works::
+
+ >>> sel.xpath("//a[contains(., 'Next Page')]").getall()
+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
+
+.. _`XPath string function`: https://www.w3.org/TR/xpath/#section-String-Functions
+
.. _topics-selectors-xpath-variables:
Variables in XPath expressions
@@ -461,6 +632,69 @@ on `XPath variables`_.
.. _XPath variables: https://parsel.readthedocs.io/en/latest/usage.html#variables-in-xpath-expressions
+
+.. _removing-namespaces:
+
+Removing namespaces
+-------------------
+
+When dealing with scraping projects, it is often quite convenient to get rid of
+namespaces altogether and just work with element names, to write more
+simple/convenient XPaths. You can use the
+:meth:`Selector.remove_namespaces` method for that.
+
+Let's show an example that illustrates this with GitHub blog atom feed.
+
+.. highlight:: sh
+
+First, we open the shell with the url we want to scrape::
+
+ $ scrapy shell https://github.com/blog.atom
+
+.. highlight:: xml
+
+This is how the file starts::
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <feed xml:lang="en-US"
+ xmlns="http://www.w3.org/2005/Atom"
+ xmlns:media="http://search.yahoo.com/mrss/">
+ <id>tag:github.com,2008:/blog</id>
+ ...
+
+You can see two namespace declarations: a default "http://www.w3.org/2005/Atom"
+and another one using the "media:" prefix for "http://search.yahoo.com/mrss/".
+
+.. highlight:: python
+
+Once in the shell we can try selecting all ``<link>`` objects and see that it
+doesn't work (because the Atom XML namespace is obfuscating those nodes)::
+
+ >>> response.xpath("//link")
+ []
+
+But once we call the :meth:`Selector.remove_namespaces` method, all
+nodes can be accessed directly by their names::
+
+ >>> response.selector.remove_namespaces()
+ >>> response.xpath("//link")
+ [<Selector xpath='//link' data='<link xmlns="http://www.w3.org/2005/Atom'>,
+ <Selector xpath='//link' data='<link xmlns="http://www.w3.org/2005/Atom'>,
+ ...
+
+If you wonder why the namespace removal procedure isn't always called by default
+instead of having to call it manually, this is because of two reasons, which, in order
+of relevance, are:
+
+1. Removing namespaces requires to iterate and modify all nodes in the
+ document, which is a reasonably expensive operation to perform by default
+ for all documents crawled by Scrapy
+
+2. There could be some cases where using namespaces is actually required, in
+ case some element names clash between namespaces. These cases are very rare
+ though.
+
+
Using EXSLT extensions
----------------------
@@ -606,174 +840,44 @@ inside another ``itemscope``.
.. _regular expressions: http://exslt.org/regexp/index.html
.. _set manipulation: http://exslt.org/set/index.html
+Other XPath extensions
+----------------------
-Some XPath tips
----------------
-
-Here are some tips that you may find useful when using XPath
-with Scrapy selectors, based on `this post from ScrapingHub's blog`_.
-If you are not much familiar with XPath yet,
-you may want to take a look first at this `XPath tutorial`_.
-
-
-.. _`XPath tutorial`: http://www.zvon.org/comp/r/tut-XPath_1.html
-.. _`this post from ScrapingHub's blog`: https://blog.scrapinghub.com/2014/07/17/xpath-tips-from-the-web-scraping-trenches/
-
-
-Using text nodes in a condition
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When you need to use the text content as argument to an `XPath string function`_,
-avoid using ``.//text()`` and use just ``.`` instead.
-
-This is because the expression ``.//text()`` yields a collection of text elements -- a *node-set*.
-And when a node-set is converted to a string, which happens when it is passed as argument to
-a string function like ``contains()`` or ``starts-with()``, it results in the text for the first element only.
+Scrapy selectors also provide a sorely missed XPath extension function
+``has-class`` that returns ``True`` for nodes that have all of the specified
+HTML classes.
-Example::
+.. highlight:: html
- >>> from scrapy import Selector
- >>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
+For the following HTML::
-Converting a *node-set* to string::
+ <p class="foo bar-baz">First</p>
+ <p class="foo">Second</p>
+ <p class="bar">Third</p>
+ <p>Fourth</p>
- >>> sel.xpath('//a//text()').getall() # take a peek at the node-set
- ['Click here to go to the ', 'Next Page']
- >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
- ['Click here to go to the ']
-
-A *node* converted to a string, however, puts together the text of itself plus of all its descendants::
-
- >>> sel.xpath("//a[1]").getall() # select the first node
- ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
- >>> sel.xpath("string(//a[1])").getall() # convert it to string
- ['Click here to go to the Next Page']
+.. highlight:: python
-So, using the ``.//text()`` node-set won't select anything in this case::
+You can use it like this::
- >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
+ >>> response.xpath('//p[has-class("foo")]')
+ [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
+ <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
+ >>> response.xpath('//p[has-class("foo", "bar-baz")]')
+ [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
+ >>> response.xpath('//p[has-class("foo", "bar")]')
[]
-But using the ``.`` to mean the node, works::
-
- >>> sel.xpath("//a[contains(., 'Next Page')]").getall()
- ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
-
-.. _`XPath string function`: https://www.w3.org/TR/xpath/#section-String-Functions
-
-Beware of the difference between //node[1] and (//node)[1]
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-``//node[1]`` selects all the nodes occurring first under their respective parents.
-
-``(//node)[1]`` selects all the nodes in the document, and then gets only the first of them.
-
-Example::
-
- >>> from scrapy import Selector
- >>> sel = Selector(text="""
- ....: <ul class="list">
- ....: <li>1</li>
- ....: <li>2</li>
- ....: <li>3</li>
- ....: </ul>
- ....: <ul class="list">
- ....: <li>4</li>
- ....: <li>5</li>
- ....: <li>6</li>
- ....: </ul>""")
- >>> xp = lambda x: sel.xpath(x).getall()
-
-This gets all first ``<li>`` elements under whatever it is its parent::
-
- >>> xp("//li[1]")
- ['<li>1</li>', '<li>4</li>']
-
-And this gets the first ``<li>`` element in the whole document::
+So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS
+``p.foo.bar-baz``. Please note, that it is slower in most of the cases,
+because it's a pure-Python function that's invoked for every node in question
+whereas the CSS lookup is translated into XPath and thus runs more efficiently,
+so performance-wise its uses are limited to situations that are not easily
+described with CSS selectors.
- >>> xp("(//li)[1]")
- ['<li>1</li>']
+Parsel also simplifies adding your own XPath extensions.
-This gets all first ``<li>`` elements under an ``<ul>`` parent::
-
- >>> xp("//ul/li[1]")
- ['<li>1</li>', '<li>4</li>']
-
-And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document::
-
- >>> xp("(//ul/li)[1]")
- ['<li>1</li>']
-
-When querying by class, consider using CSS
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Because an element can contain multiple CSS classes, the XPath way to select elements
-by class is the rather verbose::
-
- *[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')]
-
-If you use ``@class='someclass'`` you may end up missing elements that have
-other classes, and if you just use ``contains(@class, 'someclass')`` to make up
-for that you may end up with more elements that you want, if they have a different
-class name that shares the string ``someclass``.
-
-As it turns out, Scrapy selectors allow you to chain selectors, so most of the time
-you can just select by class using CSS and then switch to XPath when needed::
-
- >>> from scrapy import Selector
- >>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')
- >>> sel.css('.shout').xpath('./time/@datetime').getall()
- ['2014-07-23 19:00']
-
-This is cleaner than using the verbose XPath trick shown above. Just remember
-to use the ``.`` in the XPath expressions that will follow.
-
-.. _old-extraction-api:
-
-extract() and extract_first()
------------------------------
-
-If you're a long-time Scrapy user, you're probably familiar
-with ``.extract()`` and ``.extract_first()`` selector methods. These methods
-are still supported by Scrapy, there are no plans to deprecate them.
-
-However, Scrapy usage docs are now written using ``.get()`` and
-``.getall()`` methods. We feel that these new methods result in a more concise
-and readable code.
-
-The following examples show how these methods map to each other.
-
-1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``::
-
- >>> response.css('a::attr(href)').get()
- 'image1.html'
- >>> response.css('a::attr(href)').extract_first()
- 'image1.html'
-
-2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``::
-
- >>> response.css('a::attr(href)').getall()
- ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
- >>> response.css('a::attr(href)').extract()
- ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
-
-2. ``Selector.get()`` is the same as ``Selector.extract()``::
-
- >>> response.css('a::attr(href)')[0].get()
- 'image1.html'
- >>> response.css('a::attr(href)')[0].extract()
- 'image1.html'
-
-4. For consistency, there is also ``Selector.getall()``, which returns a list::
-
- >>> response.css('a::attr(href)')[0].getall()
- ['image1.html']
-
-So, the main difference is that output of ``.get()`` and ``.getall()`` methods
-is more predictable: ``.get()`` always returns a single result, ``.getall()``
-always returns a list of all extracted results. With ``.extract()`` method
-it was not always obvious if a result is a list or not; to get a single
-result either ``.extract()`` or ``.extract_first()`` should be called.
+.. autofunction:: parsel.xpathfuncs.set_xpathfunc
.. _topics-selectors-ref:
@@ -909,6 +1013,11 @@ SelectorList objects
their results flattened, as a list of unicode strings.
+.. _selector-examples:
+
+Examples
+========
+
.. _selector-examples-html:
Selector examples on HTML response
@@ -958,65 +1067,4 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object::
sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall()
-.. _removing-namespaces:
-
-Removing namespaces
--------------------
-
-When dealing with scraping projects, it is often quite convenient to get rid of
-namespaces altogether and just work with element names, to write more
-simple/convenient XPaths. You can use the
-:meth:`Selector.remove_namespaces` method for that.
-
-Let's show an example that illustrates this with GitHub blog atom feed.
-
-.. highlight:: sh
-
-First, we open the shell with the url we want to scrape::
-
- $ scrapy shell https://github.com/blog.atom
-
-.. highlight:: xml
-
-This is how the file starts::
-
- <?xml version="1.0" encoding="UTF-8"?>
- <feed xml:lang="en-US"
- xmlns="http://www.w3.org/2005/Atom"
- xmlns:media="http://search.yahoo.com/mrss/">
- <id>tag:github.com,2008:/blog</id>
- ...
-
-You can see two namespace declarations: a default "http://www.w3.org/2005/Atom"
-and another one using the "media:" prefix for "http://search.yahoo.com/mrss/".
-
-.. highlight:: python
-
-Once in the shell we can try selecting all ``<link>`` objects and see that it
-doesn't work (because the Atom XML namespace is obfuscating those nodes)::
-
- >>> response.xpath("//link")
- []
-
-But once we call the :meth:`Selector.remove_namespaces` method, all
-nodes can be accessed directly by their names::
-
- >>> response.selector.remove_namespaces()
- >>> response.xpath("//link")
- [<Selector xpath='//link' data='<link xmlns="http://www.w3.org/2005/Atom'>,
- <Selector xpath='//link' data='<link xmlns="http://www.w3.org/2005/Atom'>,
- ...
-
-If you wonder why the namespace removal procedure isn't always called by default
-instead of having to call it manually, this is because of two reasons, which, in order
-of relevance, are:
-
-1. Removing namespaces requires to iterate and modify all nodes in the
- document, which is a reasonably expensive operation to perform by default
- for all documents crawled by Scrapy
-
-2. There could be some cases where using namespaces is actually required, in
- case some element names clash between namespaces. These cases are very rare
- though.
-
.. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799