Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Format python console blocks using blacken-docs

pankaj1707kcc9eb3fa7961
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f5e9b372e..e565c3ebb 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -83,7 +83,9 @@ optionally how to follow links in the pages, and how to parse the downloaded
page content to extract data.
This is the code for our first Spider. Save it in a file named
-``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::
+``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:
+
+.. code-block:: python
from pathlib import Path
@@ -95,17 +97,17 @@ This is the code for our first Spider. Save it in a file named
def start_requests(self):
urls = [
- 'https://quotes.toscrape.com/page/1/',
- 'https://quotes.toscrape.com/page/2/',
+ "https://quotes.toscrape.com/page/1/",
+ "https://quotes.toscrape.com/page/2/",
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
page = response.url.split("/")[-2]
- filename = f'quotes-{page}.html'
+ filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
- self.log(f'Saved file {filename}')
+ self.log(f"Saved file {filename}")
As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`
@@ -247,8 +249,10 @@ object:
response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')
->>> response.css('title')
-[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
+.. code-block:: pycon
+
+ >>> response.css("title")
+ [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
The result of running ``response.css('title')`` is a list-like object called
:class:`~scrapy.selector.SelectorList`, which represents a list of
@@ -258,42 +262,54 @@ data.
To extract the text from the title above, you can do:
->>> response.css('title::text').getall()
-['Quotes to Scrape']
+.. code-block:: pycon
+
+ >>> response.css("title::text").getall()
+ ['Quotes to Scrape']
There are two things to note here: one is that we've added ``::text`` to the
CSS query, to mean we want to select only the text elements directly inside
``<title>`` element. If we don't specify ``::text``, we'd get the full title
element, including its tags:
->>> response.css('title').getall()
-['<title>Quotes to Scrape</title>']
+.. code-block:: pycon
+
+ >>> response.css("title").getall()
+ ['<title>Quotes to Scrape</title>']
The other thing is that the result of calling ``.getall()`` is a list: it is
possible that a selector returns more than one result, so we extract them all.
When you know you just want the first result, as in this case, you can do:
->>> response.css('title::text').get()
-'Quotes to Scrape'
+.. code-block:: pycon
+
+ >>> response.css("title::text").get()
+ 'Quotes to Scrape'
As an alternative, you could've written:
->>> response.css('title::text')[0].get()
-'Quotes to Scrape'
+.. code-block:: pycon
+
+ >>> response.css("title::text")[0].get()
+ 'Quotes to Scrape'
Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will
-raise an :exc:`IndexError` exception if there are no results::
+raise an :exc:`IndexError` exception if there are no results:
+
+.. code-block:: pycon
- >>> response.css('noelement')[0].get()
+ >>> response.css("noelement")[0].get()
Traceback (most recent call last):
...
IndexError: list index out of range
You might want to use ``.get()`` directly on the
:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``
-if there are no results::
+if there are no results:
->>> response.css("noelement").get()
+.. code-block:: pycon
+
+ >>> response.css("noelement").get()
There's a lesson here: for most scraping code, you want it to be resilient to
errors due to things not being found on a page, so that even if some parts fail
@@ -304,12 +320,14 @@ Besides the :meth:`~scrapy.selector.SelectorList.getall` and
the :meth:`~scrapy.selector.SelectorList.re` method to extract using
:doc:`regular expressions <library/re>`:
->>> response.css('title::text').re(r'Quotes.*')
-['Quotes to Scrape']
->>> response.css('title::text').re(r'Q\w+')
-['Quotes']
->>> response.css('title::text').re(r'(\w+) to (\w+)')
-['Quotes', 'Scrape']
+.. code-block:: pycon
+
+ >>> response.css("title::text").re(r"Quotes.*")
+ ['Quotes to Scrape']
+ >>> response.css("title::text").re(r"Q\w+")
+ ['Quotes']
+ >>> response.css("title::text").re(r"(\w+) to (\w+)")
+ ['Quotes', 'Scrape']
In order to find the proper CSS selectors to use, you might find useful opening
the response page from the shell in your web browser using ``view(response)``.
@@ -327,10 +345,12 @@ XPath: a brief intro
Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions:
->>> response.xpath('//title')
-[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
->>> response.xpath('//title/text()').get()
-'Quotes to Scrape'
+.. code-block:: pycon
+
+ >>> response.xpath("//title")
+ [<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
+ >>> response.xpath("//title/text()").get()
+ 'Quotes to Scrape'
XPath expressions are very powerful, and are the foundation of Scrapy
Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
@@ -387,33 +407,41 @@ we want::
We get a list of selectors for the quote HTML elements with:
->>> response.css("div.quote")
-[<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
- <Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
- ...]
+.. code-block:: pycon
+
+ >>> response.css("div.quote")
+ [<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
+ <Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,
+ ...]
Each of the selectors returned by the query above allows us to run further
queries over their sub-elements. Let's assign the first selector to a
variable, so that we can run our CSS selectors directly on a particular quote:
->>> quote = response.css("div.quote")[0]
+.. code-block:: pycon
+
+ >>> quote = response.css("div.quote")[0]
Now, let's extract ``text``, ``author`` and the ``tags`` from that quote
using the ``quote`` object we just created:
->>> text = quote.css("span.text::text").get()
->>> text
-'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
->>> author = quote.css("small.author::text").get()
->>> author
-'Albert Einstein'
+.. code-block:: pycon
+
+ >>> text = quote.css("span.text::text").get()
+ >>> text
+ '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
+ >>> author = quote.css("small.author::text").get()
+ >>> author
+ 'Albert Einstein'
Given that the tags are a list of strings, we can use the ``.getall()`` method
to get all of them:
->>> tags = quote.css("div.tags a.tag::text").getall()
->>> tags
-['change', 'deep-thoughts', 'thinking', 'world']
+.. code-block:: pycon
+
+ >>> tags = quote.css("div.tags a.tag::text").getall()
+ >>> tags
+ ['change', 'deep-thoughts', 'thinking', 'world']
.. invisible-code-block: python
@@ -422,14 +450,17 @@ to get all of them:
Having figured out how to extract each bit, we can now iterate over all the
quotes elements and put them together into a Python dictionary:
->>> for quote in response.css("div.quote"):
-... text = quote.css("span.text::text").get()
-... author = quote.css("small.author::text").get()
-... tags = quote.css("div.tags a.tag::text").getall()
-... print(dict(text=text, author=author, tags=tags))
-{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
-{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}
-...
+.. code-block:: pycon
+
+ >>> for quote in response.css("div.quote"):
+ ... text = quote.css("span.text::text").get()
+ ... author = quote.css("small.author::text").get()
+ ... tags = quote.css("div.tags a.tag::text").getall()
+ ... print(dict(text=text, author=author, tags=tags))
+ ...
+ {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
+ {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}
+ ...
Extracting data in our spider
-----------------------------
@@ -537,14 +568,18 @@ This gets the anchor element, but we want the attribute ``href``. For that,
Scrapy supports a CSS extension that lets you select the attribute contents,
like this:
->>> response.css('li.next a::attr(href)').get()
-'/page/2/'
+.. code-block:: pycon
+
+ >>> response.css("li.next a::attr(href)").get()
+ '/page/2/'
There is also an ``attrib`` property available
(see :ref:`selecting-attributes` for more):
->>> response.css('li.next a').attrib['href']
-'/page/2/'
+.. code-block:: pycon
+
+ >>> response.css("li.next a").attrib["href"]
+ '/page/2/'
Let's see now our spider modified to recursively follow the link to the next
page, extracting data from it:
diff --git a/docs/news.rst b/docs/news.rst
index 07264827b..dd1574e26 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -2417,11 +2417,13 @@ Backward-incompatible changes
* :class:`~scrapy.loader.ItemLoader` now turns the values of its input item
into lists:
- >>> item = MyItem()
- >>> item['field'] = 'value1'
- >>> loader = ItemLoader(item=item)
- >>> item['field']
- ['value1']
+ .. code-block:: pycon
+
+ >>> item = MyItem()
+ >>> item["field"] = "value1"
+ >>> loader = ItemLoader(item=item)
+ >>> item["field"]
+ ['value1']
This is needed to allow adding values to existing fields
(``loader.add_value('field', 'value2')``).
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 60b5acd10..1eac444e6 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -134,14 +134,14 @@ Settings API
.. highlight:: python
- ::
+ .. code-block:: python
SETTINGS_PRIORITIES = {
- 'default': 0,
- 'command': 10,
- 'project': 20,
- 'spider': 30,
- 'cmdline': 40,
+ "default": 0,
+ "command": 10,
+ "project": 20,
+ "spider": 30,
+ "cmdline": 40,
}
For a detailed explanation on each settings sources, see:
diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst
index 39e7b7d3c..a15ee1059 100644
--- a/docs/topics/developer-tools.rst
+++ b/docs/topics/developer-tools.rst
@@ -94,8 +94,10 @@ Then, back to your web browser, right-click on the ``span`` tag, select
response = load_response('https://quotes.toscrape.com/', 'quotes.html')
->>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall()
-['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
+.. code-block:: pycon
+
+ >>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall()
+ ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
Adding ``text()`` at the end we are able to extract the first quote with this
basic selector. But this XPath is not really that clever. All it does is
@@ -124,11 +126,13 @@ With this knowledge we can refine our XPath: Instead of a path to follow,
we'll simply select all ``span`` tags with the ``class="text"`` by using
the `has-class-extension`_:
->>> response.xpath('//span[has-class("text")]/text()').getall()
-['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
-'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
-'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
-...]
+.. code-block:: pycon
+
+ >>> response.xpath('//span[has-class("text")]/text()').getall()
+ ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
+ '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
+ '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
+ ...]
And with one simple, cleverer XPath we are able to extract all quotes from
the page. We could have constructed a loop over our first XPath to increase
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst
index 9be0ed058..d01e0a8d4 100644
--- a/docs/topics/dynamic-content.rst
+++ b/docs/topics/dynamic-content.rst
@@ -183,10 +183,12 @@ data from it:
For example, if the JavaScript code contains a separate line like
``var data = {"field": "value"};`` you can extract that data as follows:
- >>> pattern = r'\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n'
- >>> json_data = response.css('script::text').re_first(pattern)
- >>> json.loads(json_data)
- {'field': 'value'}
+ .. code-block:: pycon
+
+ >>> pattern = r"\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n"
+ >>> json_data = response.css("script::text").re_first(pattern)
+ >>> json.loads(json_data)
+ {'field': 'value'}
- chompjs_ provides an API to parse JavaScript objects into a :class:`dict`.
@@ -194,11 +196,13 @@ data from it:
``var data = {field: "value", secondField: "second value"};``
you can extract that data as follows:
- >>> import chompjs
- >>> javascript = response.css('script::text').get()
- >>> data = chompjs.parse_js_object(javascript)
- >>> data
- {'field': 'value', 'secondField': 'second value'}
+ .. code-block:: pycon
+
+ >>> import chompjs
+ >>> javascript = response.css("script::text").get()
+ >>> data = chompjs.parse_js_object(javascript)
+ >>> data
+ {'field': 'value', 'secondField': 'second value'}
- Otherwise, use js2xml_ to convert the JavaScript code into an XML document
that you can parse using :ref:`selectors <topics-selectors>`.
@@ -206,14 +210,16 @@ data from it:
For example, if the JavaScript code contains
``var data = {field: "value"};`` you can extract that data as follows:
- >>> import js2xml
- >>> import lxml.etree
- >>> from parsel import Selector
- >>> javascript = response.css('script::text').get()
- >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding='unicode')
- >>> selector = Selector(text=xml)
- >>> selector.css('var[name="data"]').get()
- '<var name="data"><object><property name="field"><string>value</string></property></object></var>'
+ .. code-block:: pycon
+
+ >>> import js2xml
+ >>> import lxml.etree
+ >>> from parsel import Selector
+ >>> javascript = response.css("script::text").get()
+ >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding="unicode")
+ >>> selector = Selector(text=xml)
+ >>> selector.css('var[name="data"]').get()
+ '<var name="data"><object><property name="field"><string>value</string></property></object></var>'
.. _topics-javascript-rendering:
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index 7b6dfc5c1..3c38ac2dc 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -234,62 +234,68 @@ notice the API is very similar to the :class:`dict` API.
Creating items
''''''''''''''
->>> product = Product(name='Desktop PC', price=1000)
->>> print(product)
-Product(name='Desktop PC', price=1000)
+.. code-block:: pycon
+
+ >>> product = Product(name="Desktop PC", price=1000)
+ >>> print(product)
+ Product(name='Desktop PC', price=1000)
Getting field values
''''''''''''''''''''
->>> product['name']
-Desktop PC
->>> product.get('name')
-Desktop PC
+.. code-block:: pycon
+
+ >>> product["name"]
+ Desktop PC
+ >>> product.get("name")
+ Desktop PC
->>> product['price']
-1000
+ >>> product["price"]
+ 1000
->>> product['last_updated']
-Traceback (most recent call last):
- ...
-KeyError: 'last_updated'
+ >>> product["last_updated"]
+ Traceback (most recent call last):
+ ...
+ KeyError: 'last_updated'
->>> product.get('last_updated', 'not set')
-not set
+ >>> product.get("last_updated", "not set")
+ not set
->>> product['lala'] # getting unknown field
-Traceback (most recent call last):
- ...
-KeyError: 'lala'
+ >>> product["lala"] # getting unknown field
+ Traceback (most recent call last):
+ ...
+ KeyError: 'lala'
->>> product.get('lala', 'unknown field')
-'unknown field'
+ >>> product.get("lala", "unknown field")
+ 'unknown field'
->>> 'name' in product # is name field populated?
-True
+ >>> "name" in product # is name field populated?
+ True
->>> 'last_updated' in product # is last_updated populated?
-False
+ >>> "last_updated" in product # is last_updated populated?
+ False
->>> 'last_updated' in product.fields # is last_updated a declared field?
-True
+ >>> "last_updated" in product.fields # is last_updated a declared field?
+ True
->>> 'lala' in product.fields # is lala a declared field?
-False
+ >>> "lala" in product.fields # is lala a declared field?
+ False
Setting field values
''''''''''''''''''''
->>> product['last_updated'] = 'today'
->>> product['last_updated']
-today
+.. code-block:: pycon
+
+ >>> product["last_updated"] = "today"
+ >>> product["last_updated"]
+ today
->>> product['lala'] = 'test' # setting unknown field
-Traceback (most recent call last):
- ...
-KeyError: 'Product does not support field: lala'
+ >>> product["lala"] = "test" # setting unknown field
+ Traceback (most recent call last):
+ ...
+ KeyError: 'Product does not support field: lala'
Accessing all populated values
@@ -297,11 +303,13 @@ Accessing all populated values
To access all populated values, just use the typical :class:`dict` API:
->>> product.keys()
-['price', 'name']
+.. code-block:: pycon
->>> product.items()
-[('price', 1000), ('name', 'Desktop PC')]
+ >>> product.keys()
+ ['price', 'name']
+
+ >>> product.items()
+ [('price', 1000), ('name', 'Desktop PC')]
.. _copying-items:
@@ -339,18 +347,20 @@ Other common tasks
Creating dicts from items:
->>> dict(product) # create a dict from all populated values
-{'price': 1000, 'name': 'Desktop PC'}
+.. code-block:: pycon
+
+ >>> dict(product) # create a dict from all populated values
+ {'price': 1000, 'name': 'Desktop PC'}
-Creating items from dicts:
+ Creating items from dicts:
->>> Product({'name': 'Laptop PC', 'price': 1500})
-Product(price=1500, name='Laptop PC')
+ >>> Product({"name": "Laptop PC", "price": 1500})
+ Product(price=1500, name='Laptop PC')
->>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict
-Traceback (most recent call last):
- ...
-KeyError: 'Product does not support field: lala'
+ >>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict
+ Traceback (most recent call last):
+ ...
+ KeyError: 'Product does not support field: lala'
Extending Item subclasses
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index 33441838a..cd8914644 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -70,13 +70,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function::
telnet localhost 6023
- >>> prefs()
- Live References
+ .. code-block:: pycon
- ExampleSpider 1 oldest: 15s ago
- HtmlResponse 10 oldest: 1s ago
- Selector 2 oldest: 0s ago
- FormRequest 878 oldest: 7s ago
+ >>> prefs()
+ Live References
+
+ ExampleSpider 1 oldest: 15s ago
+ HtmlResponse 10 oldest: 1s ago
+ Selector 2 oldest: 0s ago
+ FormRequest 878 oldest: 7s ago
As you can see, that report also shows the "age" of the oldest object in each
class. If you're running multiple spiders per process chances are you can
@@ -114,7 +116,9 @@ a priori, of course) by using the ``trackref`` tool.
After the crawler is running for a few minutes and we notice its memory usage
has grown a lot, we can enter its telnet console and check the live
-references::
+references:
+
+.. code-block:: pycon
>>> prefs()
Live References
@@ -134,19 +138,23 @@ generating the leaks (passing response references inside requests).
Sometimes extra information about live objects can be helpful.
Let's check the oldest response:
->>> from scrapy.utils.trackref import get_oldest
->>> r = get_oldest('HtmlResponse')
->>> r.url
-'http://www.somenastyspider.com/product.php?pid=123'
+.. code-block:: pycon
+
+ >>> from scrapy.utils.trackref import get_oldest
+ >>> r = get_oldest("HtmlResponse")
+ >>> r.url
+ 'http://www.somenastyspider.com/product.php?pid=123'
If you want to iterate over all objects, instead of getting the oldest one, you
can use the :func:`scrapy.utils.trackref.iter_all` function:
->>> from scrapy.utils.trackref import iter_all
->>> [r.url for r in iter_all('HtmlResponse')]
-['http://www.somenastyspider.com/product.php?pid=123',
- 'http://www.somenastyspider.com/product.php?pid=584',
-...]
+.. code-block:: pycon
+
+ >>> from scrapy.utils.trackref import iter_all
+ >>> [r.url for r in iter_all("HtmlResponse")]
+ ['http://www.somenastyspider.com/product.php?pid=123',
+ 'http://www.somenastyspider.com/product.php?pid=584',
+ ...]
Too many spiders?
-----------------
@@ -157,8 +165,10 @@ For this reason, that function has a ``ignore`` argument which can be used to
ignore a particular class (and all its subclasses). For
example, this won't show any live references to spiders:
->>> from scrapy.spiders import Spider
->>> prefs(ignore=Spider)
+.. code-block:: pycon
+
+ >>> from scrapy.spiders import Spider
+ >>> prefs(ignore=Spider)
.. module:: scrapy.utils.trackref
:synopsis: Track references of live objects
@@ -216,30 +226,32 @@ If you use ``pip``, you can install muppy with the following command::
Here's an example to view all Python objects available in
the heap using muppy:
->>> from pympler import muppy
->>> all_objects = muppy.get_objects()
->>> len(all_objects)
-28667
->>> from pympler import summary
->>> suml = summary.summarize(all_objects)
->>> summary.print_(suml)
- types | # objects | total size
-==================================== | =========== | ============
- <class 'str | 9822 | 1.10 MB
- <class 'dict | 1658 | 856.62 KB
- <class 'type | 436 | 443.60 KB
- <class 'code | 2974 | 419.56 KB
- <class '_io.BufferedWriter | 2 | 256.34 KB
- <class 'set | 420 | 159.88 KB
- <class '_io.BufferedReader | 1 | 128.17 KB
- <class 'wrapper_descriptor | 1130 | 88.28 KB
- <class 'tuple | 1304 | 86.57 KB
- <class 'weakref | 1013 | 79.14 KB
- <class 'builtin_function_or_method | 958 | 67.36 KB
- <class 'method_descriptor | 865 | 60.82 KB
- <class 'abc.ABCMeta | 62 | 59.96 KB
- <class 'list | 446 | 58.52 KB
- <class 'int | 1425 | 43.20 KB
+.. code-block:: pycon
+
+ >>> from pympler import muppy
+ >>> all_objects = muppy.get_objects()
+ >>> len(all_objects)
+ 28667
+ >>> from pympler import summary
+ >>> suml = summary.summarize(all_objects)
+ >>> summary.print_(suml)
+ types | # objects | total size
+ ==================================== | =========== | ============
+ <class 'str | 9822 | 1.10 MB
+ <class 'dict | 1658 | 856.62 KB
+ <class 'type | 436 | 443.60 KB
+ <class 'code | 2974 | 419.56 KB
+ <class '_io.BufferedWriter | 2 | 256.34 KB
+ <class 'set | 420 | 159.88 KB
+ <class '_io.BufferedReader | 1 | 128.17 KB
+ <class 'wrapper_descriptor | 1130 | 88.28 KB
+ <class 'tuple | 1304 | 86.57 KB
+ <class 'weakref | 1013 | 79.14 KB
+ <class 'builtin_function_or_method | 958 | 67.36 KB
+ <class 'method_descriptor | 865 | 60.82 KB
+ <class 'abc.ABCMeta | 62 | 59.96 KB
+ <class 'list | 446 | 58.52 KB
+ <class 'int | 1425 | 43.20 KB
For more info about muppy, refer to the `muppy documentation`_.
diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst
index eaad926d1..45e19ef07 100644
--- a/docs/topics/loaders.rst
+++ b/docs/topics/loaders.rst
@@ -250,12 +250,15 @@ metadata. Here is an example:
output_processor=TakeFirst(),
)
->>> from scrapy.loader import ItemLoader
->>> il = ItemLoader(item=Product())
->>> il.add_value('name', ['Welcome to my', '<strong>website</strong>'])
->>> il.add_value('price', ['&euro;', '<span>1000</span>'])
->>> il.load_item()
-{'name': 'Welcome to my website', 'price': '1000'}
+
+.. code-block:: pycon
+
+ >>> from scrapy.loader import ItemLoader
+ >>> il = ItemLoader(item=Product())
+ >>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])
+ >>> il.add_value("price", ["&euro;", "<span>1000</span>"])
+ >>> il.load_item()
+ {'name': 'Welcome to my website', 'price': '1000'}
The precedence order, for both input and output processors, is as follows:
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 090554e6e..fd38eb504 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -1243,8 +1243,10 @@ TextResponse objects
``str(response.body)`` is not a correct way to convert the response
body into a string:
- >>> str(b'body')
- "b'body'"
+ .. code-block:: pycon
+
+ >>> str(b"body")
+ "b'body'"
.. attribute:: TextResponse.encoding
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index ede3ca7cd..80a979650 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -51,16 +51,20 @@ Constructing selectors
Response objects expose a :class:`~scrapy.Selector` instance
on ``.selector`` attribute:
->>> response.selector.xpath('//span/text()').get()
-'good'
+.. code-block:: pycon
+
+ >>> response.selector.xpath("//span/text()").get()
+ 'good'
Querying responses using XPath and CSS is so common that responses include two
more shortcuts: ``response.xpath()`` and ``response.css()``:
->>> response.xpath('//span/text()').get()
-'good'
->>> response.css('span::text').get()
-'good'
+.. code-block:: pycon
+
+ >>> response.xpath("//span/text()").get()
+ 'good'
+ >>> response.css("span::text").get()
+ 'good'
Scrapy selectors are instances of :class:`~scrapy.Selector` class
constructed by passing either :class:`~scrapy.http.TextResponse` object or
@@ -75,19 +79,23 @@ you can also ensure the response body is parsed only once.
But if required, it is possible to use ``Selector`` directly.
Constructing from text:
->>> from scrapy.selector import Selector
->>> body = '<html><body><span>good</span></body></html>'
->>> Selector(text=body).xpath('//span/text()').get()
-'good'
+.. code-block:: pycon
+
+ >>> from scrapy.selector import Selector
+ >>> body = "<html><body><span>good</span></body></html>"
+ >>> Selector(text=body).xpath("//span/text()").get()
+ 'good'
Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of
:class:`~scrapy.http.TextResponse` subclasses:
->>> from scrapy.selector import Selector
->>> from scrapy.http import HtmlResponse
->>> response = HtmlResponse(url='http://example.com', body=body)
->>> Selector(response=response).xpath('//span/text()').get()
-'good'
+.. code-block:: pycon
+
+ >>> from scrapy.selector import Selector
+ >>> from scrapy.http import HtmlResponse
+ >>> response = HtmlResponse(url="http://example.com", body=body)
+ >>> Selector(response=response).xpath("//span/text()").get()
+ 'good'
``Selector`` automatically chooses the best parsing rules
(XML vs HTML) based on input type.
@@ -124,16 +132,20 @@ Since we're dealing with HTML, the selector will automatically use an HTML parse
So, by looking at the :ref:`HTML code <topics-selectors-htmlcode>` of that
page, let's construct an XPath for selecting the text inside the title tag:
->>> response.xpath('//title/text()')
-[<Selector xpath='//title/text()' data='Example website'>]
+.. code-block:: pycon
+
+ >>> response.xpath("//title/text()")
+ [<Selector xpath='//title/text()' data='Example website'>]
To actually extract the textual data, you must call the selector ``.get()``
or ``.getall()`` methods, as follows:
->>> response.xpath('//title/text()').getall()
-['Example website']
->>> response.xpath('//title/text()').get()
-'Example website'
+.. code-block:: pycon
+
+ >>> response.xpath("//title/text()").getall()
+ ['Example website']
+ >>> response.xpath("//title/text()").get()
+ 'Example website'
``.get()`` always returns a single result; if there are several matches,
content of a first match is returned; if there are no matches, None
@@ -142,98 +154,116 @@ is returned. ``.getall()`` returns a list with all results.
Notice that CSS selectors can select text or attribute nodes using CSS3
pseudo-elements:
->>> response.css('title::text').get()
-'Example website'
+.. code-block:: pycon
+
+ >>> response.css("title::text").get()
+ 'Example website'
As you can see, ``.xpath()`` and ``.css()`` methods return a
:class:`~scrapy.selector.SelectorList` instance, which is a list of new
selectors. This API can be used for quickly selecting nested data:
->>> response.css('img').xpath('@src').getall()
-['image1_thumb.jpg',
- 'image2_thumb.jpg',
- 'image3_thumb.jpg',
- 'image4_thumb.jpg',
- 'image5_thumb.jpg']
+.. code-block:: pycon
+
+ >>> response.css("img").xpath("@src").getall()
+ ['image1_thumb.jpg',
+ 'image2_thumb.jpg',
+ 'image3_thumb.jpg',
+ 'image4_thumb.jpg',
+ 'image5_thumb.jpg']
If you want to extract only the first matched element, you can call the
selector ``.get()`` (or its alias ``.extract_first()`` commonly used in
previous Scrapy versions):
->>> response.xpath('//div[@id="images"]/a/text()').get()
-'Name: My image 1 '
+.. code-block:: pycon
+
+ >>> response.xpath('//div[@id="images"]/a/text()').get()
+ 'Name: My image 1 '
It returns ``None`` if no element was found:
->>> response.xpath('//div[@id="not-exists"]/text()').get() is None
-True
+.. code-block:: pycon
+
+ >>> response.xpath('//div[@id="not-exists"]/text()').get() is None
+ True
A default return value can be provided as an argument, to be used instead
of ``None``:
->>> response.xpath('//div[@id="not-exists"]/text()').get(default='not-found')
-'not-found'
+.. code-block:: pycon
+
+ >>> response.xpath('//div[@id="not-exists"]/text()').get(default="not-found")
+ 'not-found'
Instead of using e.g. ``'@src'`` XPath it is possible to query for attributes
using ``.attrib`` property of a :class:`~scrapy.Selector`:
->>> [img.attrib['src'] for img in response.css('img')]
-['image1_thumb.jpg',
- 'image2_thumb.jpg',
- 'image3_thumb.jpg',
- 'image4_thumb.jpg',
- 'image5_thumb.jpg']
+.. code-block:: pycon
+
+ >>> [img.attrib["src"] for img in response.css("img")]
+ ['image1_thumb.jpg',
+ 'image2_thumb.jpg',
+ 'image3_thumb.jpg',
+ 'image4_thumb.jpg',
+ 'image5_thumb.jpg']
As a shortcut, ``.attrib`` is also available on SelectorList directly;
it returns attributes for the first matching element:
->>> response.css('img').attrib['src']
-'image1_thumb.jpg'
+.. code-block:: pycon
+
+ >>> response.css("img").attrib["src"]
+ 'image1_thumb.jpg'
This is most useful when only a single result is expected, e.g. when selecting
by id, or selecting unique elements on a web page:
->>> response.css('base').attrib['href']
-'http://example.com/'
+.. code-block:: pycon
+
+ >>> response.css("base").attrib["href"]
+ 'http://example.com/'
Now we're going to get the base URL and some image links:
->>> response.xpath('//base/@href').get()
-'http://example.com/'
-
->>> response.css('base::attr(href)').get()
-'http://example.com/'
-
->>> response.css('base').attrib['href']
-'http://example.com/'
-
->>> response.xpath('//a[contains(@href, "image")]/@href').getall()
-['image1.html',
- 'image2.html',
- 'image3.html',
- 'image4.html',
- 'image5.html']
-
->>> response.css('a[href*=image]::attr(href)').getall()
-['image1.html',
- 'image2.html',
- 'image3.html',
- 'image4.html',
- 'image5.html']
-
->>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
-['image1_thumb.jpg',
- 'image2_thumb.jpg',
- 'image3_thumb.jpg',
- 'image4_thumb.jpg',
- 'image5_thumb.jpg']
-
->>> response.css('a[href*=image] img::attr(src)').getall()
-['image1_thumb.jpg',
- 'image2_thumb.jpg',
- 'image3_thumb.jpg',
- 'image4_thumb.jpg',
- 'image5_thumb.jpg']
+.. code-block:: pycon
+
+ >>> response.xpath("//base/@href").get()
+ 'http://example.com/'
+
+ >>> response.css("base::attr(href)").get()
+ 'http://example.com/'
+
+ >>> response.css("base").attrib["href"]
+ 'http://example.com/'
+
+ >>> response.xpath('//a[contains(@href, "image")]/@href').getall()
+ ['image1.html',
+ 'image2.html',
+ 'image3.html',
+ 'image4.html',
+ 'image5.html']
+
+ >>> response.css("a[href*=image]::attr(href)").getall()
+ ['image1.html',
+ 'image2.html',
+ 'image3.html',
+ 'image4.html',
+ 'image5.html']
+
+ >>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()
+ ['image1_thumb.jpg',
+ 'image2_thumb.jpg',
+ 'image3_thumb.jpg',
+ 'image4_thumb.jpg',
+ 'image5_thumb.jpg']
+
+ >>> response.css("a[href*=image] img::attr(src)").getall()
+ ['image1_thumb.jpg',
+ 'image2_thumb.jpg',
+ 'image3_thumb.jpg',
+ 'image4_thumb.jpg',
+ 'image5_thumb.jpg']
.. _topics-selectors-css-extensions:
@@ -260,45 +290,55 @@ Examples:
* ``title::text`` selects children text nodes of a descendant ``<title>`` element:
->>> response.css('title::text').get()
-'Example website'
+.. code-block:: pycon
+
+ >>> response.css("title::text").get()
+ 'Example website'
* ``*::text`` selects all descendant text nodes of the current selector context:
->>> response.css('#images *::text').getall()
-['\n ',
- 'Name: My image 1 ',
- '\n ',
- 'Name: My image 2 ',
- '\n ',
- 'Name: My image 3 ',
- '\n ',
- 'Name: My image 4 ',
- '\n ',
- 'Name: My image 5 ',
- '\n ']
+.. code-block:: pycon
+
+ >>> response.css("#images *::text").getall()
+ ['\n ',
+ 'Name: My image 1 ',
+ '\n ',
+ 'Name: My image 2 ',
+ '\n ',
+ 'Name: My image 3 ',
+ '\n ',
+ 'Name: My image 4 ',
+ '\n ',
+ 'Name: My image 5 ',
+ '\n ']
* ``foo::text`` returns no results if ``foo`` element exists, but contains
no text (i.e. text is empty):
->>> response.css('img::text').getall()
-[]
+.. code-block:: pycon
+
+ >>> response.css("img::text").getall()
+ []
+
+ is means ``.css('foo::text').get()`` could return None even if an element
+ ists. Use ``default=''`` if you always want a string:
- This means ``.css('foo::text').get()`` could return None even if an element
- exists. Use ``default=''`` if you always want a string:
+.. code-block:: pycon
->>> response.css('img::text').get()
->>> response.css('img::text').get(default='')
-''
+ >>> response.css("img::text").get()
+ >>> response.css("img::text").get(default="")
+ ''
* ``a::attr(href)`` selects the *href* attribute value of descendant links:
->>> response.css('a::attr(href)').getall()
-['image1.html',
- 'image2.html',
- 'image3.html',
- 'image4.html',
- 'image5.html']
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)").getall()
+ ['image1.html',
+ 'image2.html',
+ 'image3.html',
+ 'image4.html',
+ 'image5.html']
.. note::
See also: :ref:`selecting-attributes`.
@@ -319,23 +359,26 @@ The selection methods (``.xpath()`` or ``.css()``) return a list of selectors
of the same type, so you can call the selection methods for those selectors
too. Here's an example:
->>> links = response.xpath('//a[contains(@href, "image")]')
->>> links.getall()
-['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
- '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
- '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
- '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
- '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
-
->>> for index, link in enumerate(links):
-... href_xpath = link.xpath('@href').get()
-... img_xpath = link.xpath('img/@src').get()
-... print(f'Link number {index} points to url {href_xpath!r} and image {img_xpath!r}')
-Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
-Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
-Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
-Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
-Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'
+.. code-block:: pycon
+
+ >>> links = response.xpath('//a[contains(@href, "image")]')
+ >>> links.getall()
+ ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
+ '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
+ '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
+ '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',
+ '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']
+
+ >>> for index, link in enumerate(links):
+ ... href_xpath = link.xpath("@href").get()
+ ... img_xpath = link.xpath("img/@src").get()
+ ... print(f"Link number {index} points to url {href_xpath!r} and image {img_xpath!r}")
+ ...
+ Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'
+ Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'
+ Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'
+ Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'
+ Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'
.. _selecting-attributes:
@@ -345,8 +388,10 @@ Selecting element attributes
There are several ways to get a value of an attribute. First, one can use
XPath syntax:
->>> response.xpath("//a/@href").getall()
-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
+.. code-block:: pycon
+
+ >>> response.xpath("//a/@href").getall()
+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
XPath syntax has a few advantages: it is a standard XPath feature, and
``@attributes`` can be used in other parts of an XPath expression - e.g.
@@ -355,30 +400,38 @@ it is possible to filter by attribute value.
Scrapy also provides an extension to CSS selectors (``::attr(...)``)
which allows to get attribute values:
->>> response.css('a::attr(href)').getall()
-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)").getall()
+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
In addition to that, there is a ``.attrib`` property of Selector.
You can use it if you prefer to lookup attributes in Python
code, without using XPaths or CSS extensions:
->>> [a.attrib['href'] for a in response.css('a')]
-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
+.. code-block:: pycon
+
+ >>> [a.attrib["href"] for a in response.css("a")]
+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
This property is also available on SelectorList; it returns a dictionary
with attributes of a first matching element. It is convenient to use when
a selector is expected to give a single result (e.g. when selecting by element
ID, or when selecting an unique element on a page):
->>> response.css('base').attrib
-{'href': 'http://example.com/'}
->>> response.css('base').attrib['href']
-'http://example.com/'
+.. code-block:: pycon
+
+ >>> response.css("base").attrib
+ {'href': 'http://example.com/'}
+ >>> response.css("base").attrib["href"]
+ 'http://example.com/'
``.attrib`` property of an empty SelectorList is empty:
->>> response.css('foo').attrib
-{}
+.. code-block:: pycon
+
+ >>> response.css("foo").attrib
+ {}
Using selectors with regular expressions
----------------------------------------
@@ -391,19 +444,23 @@ can't construct nested ``.re()`` calls.
Here's an example used to extract image names from the :ref:`HTML code
<topics-selectors-htmlcode>` above:
->>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
-['My image 1',
- 'My image 2',
- 'My image 3',
- 'My image 4',
- 'My image 5']
+.. code-block:: pycon
+
+ >>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")
+ ['My image 1',
+ 'My image 2',
+ 'My image 3',
+ 'My image 4',
+ 'My image 5']
There's an additional helper reciprocating ``.get()`` (and its
alias ``.extract_first()``) for ``.re()``, named ``.re_first()``.
Use it to extract just the first matching string:
->>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)')
-'My image 1'
+.. code-block:: pycon
+
+ >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")
+ 'My image 1'
.. _old-extraction-api:
@@ -423,28 +480,36 @@ The following examples show how these methods map to each other.
1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``:
- >>> response.css('a::attr(href)').get()
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)").get()
'image1.html'
- >>> response.css('a::attr(href)').extract_first()
+ >>> response.css("a::attr(href)").extract_first()
'image1.html'
2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``:
- >>> response.css('a::attr(href)').getall()
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)").getall()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
- >>> response.css('a::attr(href)').extract()
+ >>> response.css("a::attr(href)").extract()
['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']
3. ``Selector.get()`` is the same as ``Selector.extract()``:
- >>> response.css('a::attr(href)')[0].get()
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)")[0].get()
'image1.html'
- >>> response.css('a::attr(href)')[0].extract()
+ >>> response.css("a::attr(href)")[0].extract()
'image1.html'
4. For consistency, there is also ``Selector.getall()``, which returns a list:
- >>> response.css('a::attr(href)')[0].getall()
+.. code-block:: pycon
+
+ >>> response.css("a::attr(href)")[0].getall()
['image1.html']
So, the main difference is that output of ``.get()`` and ``.getall()`` methods
@@ -482,24 +547,34 @@ with ``/``, that XPath will be absolute to the document and not relative to the
For example, suppose you want to extract all ``<p>`` elements inside ``<div>``
elements. First, you would get all ``<div>`` elements:
->>> divs = response.xpath('//div')
+.. code-block:: pycon
+ >>> divs = response.xpath("//div")
At first, you may be tempted to use the following approach, which is wrong, as
it actually extracts all ``<p>`` elements from the document, not only those
inside ``<div>`` elements:
->>> for p in divs.xpath('//p'): # this is wrong - gets all <p> from the whole document
-... print(p.get())
+.. code-block:: pycon
+
+ >>> for p in divs.xpath("//p"): # this is wrong - gets all <p> from the whole document
+ ... print(p.get())
+ ...
This is the proper way to do it (note the dot prefixing the ``.//p`` XPath):
->>> for p in divs.xpath('.//p'): # extracts all <p> inside
-... print(p.get())
+.. code-block:: pycon
+
+ >>> for p in divs.xpath(".//p"): # extracts all <p> inside
+ ... print(p.get())
+ ...
Another common case would be to extract all direct ``<p>`` children:
->>> for p in divs.xpath('p'):
-... print(p.get())
+.. code-block:: pycon
+
+ >>> for p in divs.xpath("p"):
+ ... print(p.get())
+ ...
For more details about relative XPaths see the `Location Paths`_ section in the
XPath specification.
@@ -522,10 +597,14 @@ class name that shares the string ``someclass``.
As it turns out, Scrapy selectors allow you to chain selectors, so most of the time
you can just select by class using CSS and then switch to XPath when needed:
->>> from scrapy import Selector
->>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')
->>> sel.css('.shout').xpath('./time/@datetime').getall()
-['2014-07-23 19:00']
+.. code-block:: pycon
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(
+ ... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>'
+ ... )
+ >>> sel.css(".shout").xpath("./time/@datetime").getall()
+ ['2014-07-23 19:00']
This is cleaner than using the verbose XPath trick shown above. Just remember
to use the ``.`` in the XPath expressions that will follow.
@@ -539,39 +618,51 @@ Beware of the difference between //node[1] and (//node)[1]
Example:
->>> from scrapy import Selector
->>> sel = Selector(text="""
-....: <ul class="list">
-....: <li>1</li>
-....: <li>2</li>
-....: <li>3</li>
-....: </ul>
-....: <ul class="list">
-....: <li>4</li>
-....: <li>5</li>
-....: <li>6</li>
-....: </ul>""")
->>> xp = lambda x: sel.xpath(x).getall()
+.. code-block:: pycon
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(
+ ... text="""
+ ... <ul class="list">
+ ... <li>1</li>
+ ... <li>2</li>
+ ... <li>3</li>
+ ... </ul>
+ ... <ul class="list">
+ ... <li>4</li>
+ ... <li>5</li>
+ ... <li>6</li>
+ ... </ul>"""
+ ... )
+ >>> xp = lambda x: sel.xpath(x).getall()
This gets all first ``<li>`` elements under whatever it is its parent:
->>> xp("//li[1]")
-['<li>1</li>', '<li>4</li>']
+.. code-block:: pycon
+
+ >>> xp("//li[1]")
+ ['<li>1</li>', '<li>4</li>']
And this gets the first ``<li>`` element in the whole document:
->>> xp("(//li)[1]")
-['<li>1</li>']
+.. code-block:: pycon
+
+ >>> xp("(//li)[1]")
+ ['<li>1</li>']
This gets all first ``<li>`` elements under an ``<ul>`` parent:
->>> xp("//ul/li[1]")
-['<li>1</li>', '<li>4</li>']
+.. code-block:: pycon
+
+ >>> xp("//ul/li[1]")
+ ['<li>1</li>', '<li>4</li>']
And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document:
->>> xp("(//ul/li)[1]")
-['<li>1</li>']
+.. code-block:: pycon
+
+ >>> xp("(//ul/li)[1]")
+ ['<li>1</li>']
Using text nodes in a condition
-------------------------------
@@ -585,32 +676,44 @@ a string function like ``contains()`` or ``starts-with()``, it results in the te
Example:
->>> from scrapy import Selector
->>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
+.. code-block:: pycon
+
+ >>> from scrapy import Selector
+ >>> sel = Selector(
+ ... text='<a href="#">Click here to go to the <strong>Next Page</strong></a>'
+ ... )
Converting a *node-set* to string:
->>> sel.xpath('//a//text()').getall() # take a peek at the node-set
-['Click here to go to the ', 'Next Page']
->>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
-['Click here to go to the ']
+.. code-block:: pycon
+
+ >>> sel.xpath("//a//text()").getall() # take a peek at the node-set
+ ['Click here to go to the ', 'Next Page']
+ >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string
+ ['Click here to go to the ']
A *node* converted to a string, however, puts together the text of itself plus of all its descendants:
->>> sel.xpath("//a[1]").getall() # select the first node
-['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
->>> sel.xpath("string(//a[1])").getall() # convert it to string
-['Click here to go to the Next Page']
+.. code-block:: pycon
+
+ >>> sel.xpath("//a[1]").getall() # select the first node
+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
+ >>> sel.xpath("string(//a[1])").getall() # convert it to string
+ ['Click here to go to the Next Page']
So, using the ``.//text()`` node-set won't select anything in this case:
->>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
-[]
+.. code-block:: pycon
+
+ >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()
+ []
But using the ``.`` to mean the node, works:
->>> sel.xpath("//a[contains(., 'Next Page')]").getall()
-['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
+.. code-block:: pycon
+
+ >>> sel.xpath("//a[contains(., 'Next Page')]").getall()
+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']
.. _`XPath string function`: https://www.w3.org/TR/xpath/all/#section-String-Functions
@@ -628,15 +731,19 @@ which are then substituted with values passed with the query.
Here's an example to match an element based on its "id" attribute value,
without hard-coding it (that was shown previously):
->>> # `$val` used in the expression, a `val` argument needs to be passed
->>> response.xpath('//div[@id=$val]/a/text()', val='images').get()
-'Name: My image 1 '
+.. code-block:: pycon
+
+ >>> # `$val` used in the expression, a `val` argument needs to be passed
+ >>> response.xpath("//div[@id=$val]/a/text()", val="images").get()
+ 'Name: My image 1 '
Here's another example, to find the "id" attribute of a ``<div>`` tag containing
five ``<a>`` children (here we pass the value ``5`` as an integer):
->>> response.xpath('//div[count(a)=$cnt]/@id', cnt=5).get()
-'images'
+.. code-block:: pycon
+
+ >>> response.xpath("//div[count(a)=$cnt]/@id", cnt=5).get()
+ 'images'
All variable references must have a binding value when calling ``.xpath()``
(otherwise you'll get a ``ValueError: XPath error:`` exception).
@@ -688,17 +795,21 @@ You can see several namespace declarations including a default
Once in the shell we can try selecting all ``<link>`` objects and see that it
doesn't work (because the Atom XML namespace is obfuscating those nodes):
->>> response.xpath("//link")
-[]
+.. code-block:: pycon
+
+ >>> response.xpath("//link")
+ []
But once we call the :meth:`Selector.remove_namespaces` method, all
nodes can be accessed directly by their names:
->>> response.selector.remove_namespaces()
->>> response.xpath("//link")
-[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
- <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
- ...
+.. code-block:: pycon
+
+ >>> response.selector.remove_namespaces()
+ >>> response.xpath("//link")
+ [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,
+ <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,
+ ...
If you wonder why the namespace removal procedure isn't always called by default
instead of having to call it manually, this is because of two reasons, which, in order
@@ -735,23 +846,25 @@ The ``test()`` function, for example, can prove quite useful when XPath's
Example selecting links in list item with a "class" attribute ending with a digit:
->>> from scrapy import Selector
->>> doc = """
-... <div>
-... <ul>
-... <li class="item-0"><a href="link1.html">first item</a></li>
-... <li class="item-1"><a href="link2.html">second item</a></li>
-... <li class="item-inactive"><a href="link3.html">third item</a></li>
-... <li class="item-1"><a href="link4.html">fourth item</a></li>
-... <li class="item-0"><a href="link5.html">fifth item</a></li>
-... </ul>
-... </div>
-... """
->>> sel = Selector(text=doc, type="html")
->>> sel.xpath('//li//@href').getall()
-['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
->>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()
-['link1.html', 'link2.html', 'link4.html', 'link5.html']
+.. code-block:: pycon
+
+ >>> from scrapy import Selector
+ >>> doc = """
+ ... <div>
+ ... <ul>
+ ... <li class="item-0"><a href="link1.html">first item</a></li>
+ ... <li class="item-1"><a href="link2.html">second item</a></li>
+ ... <li class="item-inactive"><a href="link3.html">third item</a></li>
+ ... <li class="item-1"><a href="link4.html">fourth item</a></li>
+ ... <li class="item-0"><a href="link5.html">fifth item</a></li>
+ ... </ul>
+ ... </div>
+ ... """
+ >>> sel = Selector(text=doc, type="html")
+ >>> sel.xpath("//li//@href").getall()
+ ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']
+ >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()
+ ['link1.html', 'link2.html', 'link4.html', 'link5.html']
.. warning:: C library ``libxslt`` doesn't natively support EXSLT regular
expressions so `lxml`_'s implementation uses hooks to Python's ``re`` module.
@@ -765,7 +878,9 @@ These can be handy for excluding parts of a document tree before
extracting text elements for example.
Example extracting microdata (sample content taken from https://schema.org/Product)
-with groups of itemscopes and corresponding itemprops::
+with groups of itemscopes and corresponding itemprops:
+
+.. code-block:: pycon
>>> doc = """
... <div itemscope itemtype="http://schema.org/Product">
@@ -776,19 +891,15 @@ with groups of itemscopes and corresponding itemprops::
... Rated <span itemprop="ratingValue">3.5</span>/5
... based on <span itemprop="reviewCount">11</span> customer reviews
... </div>
- ...
... <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
... <span itemprop="price">$55.00</span>
... <link itemprop="availability" href="http://schema.org/InStock" />In stock
... </div>
- ...
... Product description:
... <span itemprop="description">0.7 cubic feet countertop microwave.
... Has six preset cooking categories and convenience features like
... Add-A-Minute and Child Lock.</span>
- ...
... Customer reviews:
- ...
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
... <span itemprop="name">Not a happy camper</span> -
... by <span itemprop="author">Ellie</span>,
@@ -801,7 +912,6 @@ with groups of itemscopes and corresponding itemprops::
... <span itemprop="description">The lamp burned out and now I have to replace
... it. </span>
... </div>
- ...
... <div itemprop="review" itemscope itemtype="http://schema.org/Review">
... <span itemprop="name">Value purchase</span> -
... by <span itemprop="author">Lucas</span>,
@@ -818,13 +928,16 @@ with groups of itemscopes and corresponding itemprops::
... </div>
... """
>>> sel = Selector(text=doc, type="html")
- >>> for scope in sel.xpath('//div[@itemscope]'):
- ... print("current scope:", scope.xpath('@itemtype').getall())
- ... props = scope.xpath('''
+ >>> for scope in sel.xpath("//div[@itemscope]"):
+ ... print("current scope:", scope.xpath("@itemtype").getall())
+ ... props = scope.xpath(
+ ... """
... set:difference(./descendant::*/@itemprop,
- ... .//*[@itemscope]/*/@itemprop)''')
+ ... .//*[@itemscope]/*/@itemprop)"""
+ ... )
... print(f" properties: {props.getall()}")
... print("")
+ ...
current scope: ['http://schema.org/Product']
properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review']
@@ -876,13 +989,15 @@ For the following HTML::
You can use it like this:
->>> response.xpath('//p[has-class("foo")]')
-[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
- <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
->>> response.xpath('//p[has-class("foo", "bar-baz")]')
-[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
->>> response.xpath('//p[has-class("foo", "bar")]')
-[]
+.. code-block:: pycon
+
+ >>> response.xpath('//p[has-class("foo")]')
+ [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,
+ <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]
+ >>> response.xpath('//p[has-class("foo", "bar-baz")]')
+ [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]
+ >>> response.xpath('//p[has-class("foo", "bar")]')
+ []
So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS
``p.foo.bar-baz``. Please note, that it is slower in most of the cases,
diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst
index 69a90f620..4898843e4 100644
--- a/docs/topics/shell.rst
+++ b/docs/topics/shell.rst
@@ -191,44 +191,46 @@ all start with the ``[s]`` prefix)::
After that, we can start playing with the objects:
->>> response.xpath('//title/text()').get()
-'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
-
->>> fetch("https://old.reddit.com/")
-
->>> response.xpath('//title/text()').get()
-'reddit: the front page of the internet'
-
->>> request = request.replace(method="POST")
-
->>> fetch(request)
-
->>> response.status
-404
-
->>> from pprint import pprint
-
->>> pprint(response.headers)
-{'Accept-Ranges': ['bytes'],
- 'Cache-Control': ['max-age=0, must-revalidate'],
- 'Content-Type': ['text/html; charset=UTF-8'],
- 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
- 'Server': ['snooserv'],
- 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
- 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
- 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
- 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
- 'Vary': ['accept-encoding'],
- 'Via': ['1.1 varnish'],
- 'X-Cache': ['MISS'],
- 'X-Cache-Hits': ['0'],
- 'X-Content-Type-Options': ['nosniff'],
- 'X-Frame-Options': ['SAMEORIGIN'],
- 'X-Moose': ['majestic'],
- 'X-Served-By': ['cache-cdg8730-CDG'],
- 'X-Timer': ['S1481214079.394283,VS0,VE159'],
- 'X-Ua-Compatible': ['IE=edge'],
- 'X-Xss-Protection': ['1; mode=block']}
+.. code-block:: pycon
+
+ >>> response.xpath("//title/text()").get()
+ 'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'
+
+ >>> fetch("https://old.reddit.com/")
+
+ >>> response.xpath("//title/text()").get()
+ 'reddit: the front page of the internet'
+
+ >>> request = request.replace(method="POST")
+
+ >>> fetch(request)
+
+ >>> response.status
+ 404
+
+ >>> from pprint import pprint
+
+ >>> pprint(response.headers)
+ {'Accept-Ranges': ['bytes'],
+ 'Cache-Control': ['max-age=0, must-revalidate'],
+ 'Content-Type': ['text/html; charset=UTF-8'],
+ 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],
+ 'Server': ['snooserv'],
+ 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
+ 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
+ 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',
+ 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],
+ 'Vary': ['accept-encoding'],
+ 'Via': ['1.1 varnish'],
+ 'X-Cache': ['MISS'],
+ 'X-Cache-Hits': ['0'],
+ 'X-Content-Type-Options': ['nosniff'],
+ 'X-Frame-Options': ['SAMEORIGIN'],
+ 'X-Moose': ['majestic'],
+ 'X-Served-By': ['cache-cdg8730-CDG'],
+ 'X-Timer': ['S1481214079.394283,VS0,VE159'],
+ 'X-Ua-Compatible': ['IE=edge'],
+ 'X-Xss-Protection': ['1; mode=block']}
.. _topics-shell-inspect-response:
@@ -279,14 +281,18 @@ When you run the spider, you will get something similar to this::
Then, you can check if the extraction code is working:
->>> response.xpath('//h1[@class="fn"]')
-[]
+.. code-block:: pycon
+
+ >>> response.xpath('//h1[@class="fn"]')
+ []
Nope, it doesn't. So you can open the response in your web browser and see if
it's the response you were expecting:
->>> view(response)
-True
+.. code-block:: pycon
+
+ >>> view(response)
+ True
Finally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume the
crawling::
diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst
index f8b0c4d2a..be8ecb7a5 100644
--- a/docs/topics/stats.rst
+++ b/docs/topics/stats.rst
@@ -68,13 +68,17 @@ Set stat value only if lower than previous:
Get stat value:
->>> stats.get_value('custom_count')
-1
+.. code-block:: pycon
+
+ >>> stats.get_value("custom_count")
+ 1
Get all stats:
->>> stats.get_stats()
-{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
+.. code-block:: pycon
+
+ >>> stats.get_stats()
+ {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
Available Stats Collectors
==========================