Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Enable doc tests for selectors.rst, fix issues.

Andrey Rakhmatullincba891a66c22
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index c25c75d17..4a64d530b 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -48,6 +48,8 @@ Constructing selectors
.. highlight:: python
+.. skip: start
+
Response objects expose a :class:`~scrapy.Selector` instance
on ``.selector`` attribute:
@@ -66,6 +68,8 @@ more shortcuts: ``response.xpath()`` and ``response.css()``:
>>> response.css("span::text").get()
'good'
+.. skip: end
+
Scrapy selectors are instances of :class:`~scrapy.Selector` class
constructed by passing either :class:`~scrapy.http.TextResponse` object or
markup as a string (in ``text`` argument).
@@ -93,7 +97,7 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
>>> from scrapy.selector import Selector
>>> from scrapy.http import HtmlResponse
- >>> response = HtmlResponse(url="http://example.com", body=body)
+ >>> response = HtmlResponse(url="http://example.com", body=body, encoding="utf-8")
>>> Selector(response=response).xpath("//span/text()").get()
'good'
@@ -103,6 +107,13 @@ Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
Using selectors
---------------
+.. invisible-code-block: python
+
+ html_response = response = load_response(
+ "https://docs.scrapy.org/en/latest/_static/selectors-sample1.html",
+ "../_static/selectors-sample1.html",
+ )
+
To explain how to use the selectors we'll use the ``Scrapy shell`` (which
provides interactive testing) and an example page located in the Scrapy
documentation server:
@@ -135,7 +146,7 @@ page, let's construct an XPath for selecting the text inside the title tag:
.. code-block:: pycon
>>> response.xpath("//title/text()")
- [<Selector xpath='//title/text()' data='Example website'>]
+ [<Selector query='//title/text()' data='Example website'>]
To actually extract the textual data, you must call the selector ``.get()``
or ``.getall()`` methods, as follows:
@@ -363,11 +374,11 @@ too. Here's an example:
>>> links = response.xpath('//a[contains(@href, "image")]')
>>> links.getall()
- ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
- '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
- '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
- '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
- '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
+ ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg" alt="image1"></a>',
+ '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg" alt="image2"></a>',
+ '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg" alt="image3"></a>',
+ '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg" alt="image4"></a>',
+ '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg" alt="image5"></a>']
>>> for index, link in enumerate(links):
... href_xpath = link.xpath("@href").get()
@@ -447,11 +458,11 @@ Here's an example used to extract image names from the :ref:`HTML code
.. code-block:: pycon
>>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
- ['My image 1',
- 'My image 2',
- 'My image 3',
- 'My image 4',
- 'My image 5']
+ ['My image 1 ',
+ 'My image 2 ',
+ 'My image 3 ',
+ 'My image 4 ',
+ 'My image 5 ']
There's an additional helper reciprocating ``.get()`` (and its
alias ``.extract_first()``) for ``.re()``, named ``.re_first()``.
@@ -460,7 +471,7 @@ Use it to extract just the first matching string:
.. code-block:: pycon
>>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
- 'My image 1'
+ 'My image 1 '
.. _old-extraction-api:
@@ -761,6 +772,8 @@ on `XPath variables`_.
Removing namespaces
-------------------
+.. skip: start
+
When dealing with scraping projects, it is often quite convenient to get rid of
namespaces altogether and just work with element names, to write more
simple/convenient XPaths. You can use the
@@ -808,8 +821,8 @@ nodes can be accessed directly by their names:
>>> response.selector.remove_namespaces()
>>> response.xpath("//link")
- [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
- <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
+ [<Selector query='//link' data='<link rel="alternate" type="text/html" h'>,
+ <Selector query='//link' data='<link rel="next" type="application/atom+'>,
...
If you wonder why the namespace removal procedure isn't always called by default
@@ -824,6 +837,7 @@ of relevance, are:
case some element names clash between namespaces. These cases are very rare
though.
+.. skip: end
Using EXSLT extensions
----------------------
@@ -881,6 +895,8 @@ extracting text elements for example.
Example extracting microdata (sample content taken from https://schema.org/Product)
with groups of itemscopes and corresponding itemprops:
+.. skip: next
+
.. code-block:: pycon
>>> doc = """
@@ -977,26 +993,35 @@ Scrapy selectors also provide a sorely missed XPath extension function
``has-class`` that returns ``True`` for nodes that have all of the specified
HTML classes.
-.. highlight:: html
-
-For the following HTML::
+For the following HTML:
- <p class="foo bar-baz">First</p>
- <p class="foo">Second</p>
- <p class="bar">Third</p>
- <p>Fourth</p>
+.. code-block:: pycon
-.. highlight:: python
+ >>> from scrapy.http import HtmlResponse
+ >>> response = HtmlResponse(
+ ... url="http://example.com",
+ ... body="""
+ ... <html>
+ ... <body>
+ ... <p class="foo bar-baz">First</p>
+ ... <p class="foo">Second</p>
+ ... <p class="bar">Third</p>
+ ... <p>Fourth</p>
+ ... </body>
+ ... </html>
+ ... """,
+ ... encoding="utf-8",
+ ... )
You can use it like this:
.. code-block:: pycon
>>> response.xpath('//p[has-class("foo")]')
- [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
- <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
+ [<Selector query='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
+ <Selector query='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
>>> response.xpath('//p[has-class("foo", "bar-baz")]')
- [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
+ [<Selector query='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
>>> response.xpath('//p[has-class("foo", "bar")]')
[]
@@ -1132,6 +1157,8 @@ a :class:`~scrapy.http.HtmlResponse` object like this:
Selector examples on XML response
---------------------------------
+.. skip: start
+
Here are some examples to illustrate concepts for :class:`Selector` objects
instantiated with an :class:`~scrapy.http.XmlResponse` object:
@@ -1154,4 +1181,6 @@ instantiated with an :class:`~scrapy.http.XmlResponse` object:
sel.register_namespace("g", "http://base.google.com/ns/1.0")
sel.xpath("//g:price").getall()
+.. skip: end
+
.. _Google Base XML feed: https://support.google.com/merchants/answer/160589?hl=en&ref_topic=2473799