Scrapy, a fast high-level web crawling & scraping framework for Python.
Format python console blocks using blacken-docs
pankaj1707kcc9eb3fa7961
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex f5e9b372e..e565c3ebb 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -83,7 +83,9 @@ optionally how to follow links in the pages, and how to parse the downloadedpage content to extract data.This is the code for our first Spider. Save it in a file named-``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project::+``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:++.. code-block:: pythonfrom pathlib import Path@@ -95,17 +97,17 @@ This is the code for our first Spider. Save it in a file nameddef start_requests(self):urls = [- 'https://quotes.toscrape.com/page/1/',- 'https://quotes.toscrape.com/page/2/',+ "https://quotes.toscrape.com/page/1/",+ "https://quotes.toscrape.com/page/2/",]for url in urls:yield scrapy.Request(url=url, callback=self.parse)def parse(self, response):page = response.url.split("/")[-2]- filename = f'quotes-{page}.html'+ filename = f"quotes-{page}.html"Path(filename).write_bytes(response.body)- self.log(f'Saved file {filename}')+ self.log(f"Saved file {filename}")As you can see, our Spider subclasses :class:`scrapy.Spider <scrapy.Spider>`@@ -247,8 +249,10 @@ object:response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')->>> response.css('title')-[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]+.. code-block:: pycon++ >>> response.css("title")+ [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]The result of running ``response.css('title')`` is a list-like object called:class:`~scrapy.selector.SelectorList`, which represents a list of@@ -258,42 +262,54 @@ data.To extract the text from the title above, you can do:->>> response.css('title::text').getall()-['Quotes to Scrape']+.. code-block:: pycon++ >>> response.css("title::text").getall()+ ['Quotes to Scrape']There are two things to note here: one is that we've added ``::text`` to theCSS query, to mean we want to select only the text elements directly inside``<title>`` element. If we don't specify ``::text``, we'd get the full titleelement, including its tags:->>> response.css('title').getall()-['<title>Quotes to Scrape</title>']+.. code-block:: pycon++ >>> response.css("title").getall()+ ['<title>Quotes to Scrape</title>']The other thing is that the result of calling ``.getall()`` is a list: it ispossible that a selector returns more than one result, so we extract them all.When you know you just want the first result, as in this case, you can do:->>> response.css('title::text').get()-'Quotes to Scrape'+.. code-block:: pycon++ >>> response.css("title::text").get()+ 'Quotes to Scrape'As an alternative, you could've written:->>> response.css('title::text')[0].get()-'Quotes to Scrape'+.. code-block:: pycon++ >>> response.css("title::text")[0].get()+ 'Quotes to Scrape'Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will-raise an :exc:`IndexError` exception if there are no results::+raise an :exc:`IndexError` exception if there are no results:++.. code-block:: pycon- >>> response.css('noelement')[0].get()+ >>> response.css("noelement")[0].get()Traceback (most recent call last):...IndexError: list index out of rangeYou might want to use ``.get()`` directly on the:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``-if there are no results::+if there are no results:->>> response.css("noelement").get()+.. code-block:: pycon++ >>> response.css("noelement").get()There's a lesson here: for most scraping code, you want it to be resilient toerrors due to things not being found on a page, so that even if some parts fail@@ -304,12 +320,14 @@ Besides the :meth:`~scrapy.selector.SelectorList.getall` andthe :meth:`~scrapy.selector.SelectorList.re` method to extract using:doc:`regular expressions <library/re>`:->>> response.css('title::text').re(r'Quotes.*')-['Quotes to Scrape']->>> response.css('title::text').re(r'Q\w+')-['Quotes']->>> response.css('title::text').re(r'(\w+) to (\w+)')-['Quotes', 'Scrape']+.. code-block:: pycon++ >>> response.css("title::text").re(r"Quotes.*")+ ['Quotes to Scrape']+ >>> response.css("title::text").re(r"Q\w+")+ ['Quotes']+ >>> response.css("title::text").re(r"(\w+) to (\w+)")+ ['Quotes', 'Scrape']In order to find the proper CSS selectors to use, you might find useful openingthe response page from the shell in your web browser using ``view(response)``.@@ -327,10 +345,12 @@ XPath: a brief introBesides `CSS`_, Scrapy selectors also support using `XPath`_ expressions:->>> response.xpath('//title')-[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]->>> response.xpath('//title/text()').get()-'Quotes to Scrape'+.. code-block:: pycon++ >>> response.xpath("//title")+ [<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]+ >>> response.xpath("//title/text()").get()+ 'Quotes to Scrape'XPath expressions are very powerful, and are the foundation of ScrapySelectors. In fact, CSS selectors are converted to XPath under-the-hood. You@@ -387,33 +407,41 @@ we want::We get a list of selectors for the quote HTML elements with:->>> response.css("div.quote")-[<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,- <Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,- ...]+.. code-block:: pycon++ >>> response.css("div.quote")+ [<Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,+ <Selector xpath="descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,+ ...]Each of the selectors returned by the query above allows us to run furtherqueries over their sub-elements. Let's assign the first selector to avariable, so that we can run our CSS selectors directly on a particular quote:->>> quote = response.css("div.quote")[0]+.. code-block:: pycon++ >>> quote = response.css("div.quote")[0]Now, let's extract ``text``, ``author`` and the ``tags`` from that quoteusing the ``quote`` object we just created:->>> text = quote.css("span.text::text").get()->>> text-'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'->>> author = quote.css("small.author::text").get()->>> author-'Albert Einstein'+.. code-block:: pycon++ >>> text = quote.css("span.text::text").get()+ >>> text+ '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'+ >>> author = quote.css("small.author::text").get()+ >>> author+ 'Albert Einstein'Given that the tags are a list of strings, we can use the ``.getall()`` methodto get all of them:->>> tags = quote.css("div.tags a.tag::text").getall()->>> tags-['change', 'deep-thoughts', 'thinking', 'world']+.. code-block:: pycon++ >>> tags = quote.css("div.tags a.tag::text").getall()+ >>> tags+ ['change', 'deep-thoughts', 'thinking', 'world'].. invisible-code-block: python@@ -422,14 +450,17 @@ to get all of them:Having figured out how to extract each bit, we can now iterate over all thequotes elements and put them together into a Python dictionary:->>> for quote in response.css("div.quote"):-... text = quote.css("span.text::text").get()-... author = quote.css("small.author::text").get()-... tags = quote.css("div.tags a.tag::text").getall()-... print(dict(text=text, author=author, tags=tags))-{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}-{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}-...+.. code-block:: pycon++ >>> for quote in response.css("div.quote"):+ ... text = quote.css("span.text::text").get()+ ... author = quote.css("small.author::text").get()+ ... tags = quote.css("div.tags a.tag::text").getall()+ ... print(dict(text=text, author=author, tags=tags))+ ...+ {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}+ {'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling', 'tags': ['abilities', 'choices']}+ ...Extracting data in our spider-----------------------------@@ -537,14 +568,18 @@ This gets the anchor element, but we want the attribute ``href``. For that,Scrapy supports a CSS extension that lets you select the attribute contents,like this:->>> response.css('li.next a::attr(href)').get()-'/page/2/'+.. code-block:: pycon++ >>> response.css("li.next a::attr(href)").get()+ '/page/2/'There is also an ``attrib`` property available(see :ref:`selecting-attributes` for more):->>> response.css('li.next a').attrib['href']-'/page/2/'+.. code-block:: pycon++ >>> response.css("li.next a").attrib["href"]+ '/page/2/'Let's see now our spider modified to recursively follow the link to the nextpage, extracting data from it:diff --git a/docs/news.rst b/docs/news.rstindex 07264827b..dd1574e26 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -2417,11 +2417,13 @@ Backward-incompatible changes* :class:`~scrapy.loader.ItemLoader` now turns the values of its input iteminto lists:- >>> item = MyItem()- >>> item['field'] = 'value1'- >>> loader = ItemLoader(item=item)- >>> item['field']- ['value1']+ .. code-block:: pycon++ >>> item = MyItem()+ >>> item["field"] = "value1"+ >>> loader = ItemLoader(item=item)+ >>> item["field"]+ ['value1']This is needed to allow adding values to existing fields(``loader.add_value('field', 'value2')``).diff --git a/docs/topics/api.rst b/docs/topics/api.rstindex 60b5acd10..1eac444e6 100644--- a/docs/topics/api.rst+++ b/docs/topics/api.rst@@ -134,14 +134,14 @@ Settings API.. highlight:: python- ::+ .. code-block:: pythonSETTINGS_PRIORITIES = {- 'default': 0,- 'command': 10,- 'project': 20,- 'spider': 30,- 'cmdline': 40,+ "default": 0,+ "command": 10,+ "project": 20,+ "spider": 30,+ "cmdline": 40,}For a detailed explanation on each settings sources, see:diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex 39e7b7d3c..a15ee1059 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -94,8 +94,10 @@ Then, back to your web browser, right-click on the ``span`` tag, selectresponse = load_response('https://quotes.toscrape.com/', 'quotes.html')->>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall()-['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']+.. code-block:: pycon++ >>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall()+ ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']Adding ``text()`` at the end we are able to extract the first quote with thisbasic selector. But this XPath is not really that clever. All it does is@@ -124,11 +126,13 @@ With this knowledge we can refine our XPath: Instead of a path to follow,we'll simply select all ``span`` tags with the ``class="text"`` by usingthe `has-class-extension`_:->>> response.xpath('//span[has-class("text")]/text()').getall()-['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',-'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',-'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',-...]+.. code-block:: pycon++ >>> response.xpath('//span[has-class("text")]/text()').getall()+ ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',+ '“It is our choices, Harry, that show what we truly are, far more than our abilities.”',+ '“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',+ ...]And with one simple, cleverer XPath we are able to extract all quotes fromthe page. We could have constructed a loop over our first XPath to increasediff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex 9be0ed058..d01e0a8d4 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -183,10 +183,12 @@ data from it:For example, if the JavaScript code contains a separate line like``var data = {"field": "value"};`` you can extract that data as follows:- >>> pattern = r'\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n'- >>> json_data = response.css('script::text').re_first(pattern)- >>> json.loads(json_data)- {'field': 'value'}+ .. code-block:: pycon++ >>> pattern = r"\bvar\s+data\s*=\s*(\{.*?\})\s*;\s*\n"+ >>> json_data = response.css("script::text").re_first(pattern)+ >>> json.loads(json_data)+ {'field': 'value'}- chompjs_ provides an API to parse JavaScript objects into a :class:`dict`.@@ -194,11 +196,13 @@ data from it:``var data = {field: "value", secondField: "second value"};``you can extract that data as follows:- >>> import chompjs- >>> javascript = response.css('script::text').get()- >>> data = chompjs.parse_js_object(javascript)- >>> data- {'field': 'value', 'secondField': 'second value'}+ .. code-block:: pycon++ >>> import chompjs+ >>> javascript = response.css("script::text").get()+ >>> data = chompjs.parse_js_object(javascript)+ >>> data+ {'field': 'value', 'secondField': 'second value'}- Otherwise, use js2xml_ to convert the JavaScript code into an XML documentthat you can parse using :ref:`selectors <topics-selectors>`.@@ -206,14 +210,16 @@ data from it:For example, if the JavaScript code contains``var data = {field: "value"};`` you can extract that data as follows:- >>> import js2xml- >>> import lxml.etree- >>> from parsel import Selector- >>> javascript = response.css('script::text').get()- >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding='unicode')- >>> selector = Selector(text=xml)- >>> selector.css('var[name="data"]').get()- '<var name="data"><object><property name="field"><string>value</string></property></object></var>'+ .. code-block:: pycon++ >>> import js2xml+ >>> import lxml.etree+ >>> from parsel import Selector+ >>> javascript = response.css("script::text").get()+ >>> xml = lxml.etree.tostring(js2xml.parse(javascript), encoding="unicode")+ >>> selector = Selector(text=xml)+ >>> selector.css('var[name="data"]').get()+ '<var name="data"><object><property name="field"><string>value</string></property></object></var>'.. _topics-javascript-rendering:diff --git a/docs/topics/items.rst b/docs/topics/items.rstindex 7b6dfc5c1..3c38ac2dc 100644--- a/docs/topics/items.rst+++ b/docs/topics/items.rst@@ -234,62 +234,68 @@ notice the API is very similar to the :class:`dict` API.Creating items''''''''''''''->>> product = Product(name='Desktop PC', price=1000)->>> print(product)-Product(name='Desktop PC', price=1000)+.. code-block:: pycon++ >>> product = Product(name="Desktop PC", price=1000)+ >>> print(product)+ Product(name='Desktop PC', price=1000)Getting field values''''''''''''''''''''->>> product['name']-Desktop PC->>> product.get('name')-Desktop PC+.. code-block:: pycon++ >>> product["name"]+ Desktop PC+ >>> product.get("name")+ Desktop PC->>> product['price']-1000+ >>> product["price"]+ 1000->>> product['last_updated']-Traceback (most recent call last):- ...-KeyError: 'last_updated'+ >>> product["last_updated"]+ Traceback (most recent call last):+ ...+ KeyError: 'last_updated'->>> product.get('last_updated', 'not set')-not set+ >>> product.get("last_updated", "not set")+ not set->>> product['lala'] # getting unknown field-Traceback (most recent call last):- ...-KeyError: 'lala'+ >>> product["lala"] # getting unknown field+ Traceback (most recent call last):+ ...+ KeyError: 'lala'->>> product.get('lala', 'unknown field')-'unknown field'+ >>> product.get("lala", "unknown field")+ 'unknown field'->>> 'name' in product # is name field populated?-True+ >>> "name" in product # is name field populated?+ True->>> 'last_updated' in product # is last_updated populated?-False+ >>> "last_updated" in product # is last_updated populated?+ False->>> 'last_updated' in product.fields # is last_updated a declared field?-True+ >>> "last_updated" in product.fields # is last_updated a declared field?+ True->>> 'lala' in product.fields # is lala a declared field?-False+ >>> "lala" in product.fields # is lala a declared field?+ FalseSetting field values''''''''''''''''''''->>> product['last_updated'] = 'today'->>> product['last_updated']-today+.. code-block:: pycon++ >>> product["last_updated"] = "today"+ >>> product["last_updated"]+ today->>> product['lala'] = 'test' # setting unknown field-Traceback (most recent call last):- ...-KeyError: 'Product does not support field: lala'+ >>> product["lala"] = "test" # setting unknown field+ Traceback (most recent call last):+ ...+ KeyError: 'Product does not support field: lala'Accessing all populated values@@ -297,11 +303,13 @@ Accessing all populated valuesTo access all populated values, just use the typical :class:`dict` API:->>> product.keys()-['price', 'name']+.. code-block:: pycon->>> product.items()-[('price', 1000), ('name', 'Desktop PC')]+ >>> product.keys()+ ['price', 'name']++ >>> product.items()+ [('price', 1000), ('name', 'Desktop PC')].. _copying-items:@@ -339,18 +347,20 @@ Other common tasksCreating dicts from items:->>> dict(product) # create a dict from all populated values-{'price': 1000, 'name': 'Desktop PC'}+.. code-block:: pycon++ >>> dict(product) # create a dict from all populated values+ {'price': 1000, 'name': 'Desktop PC'}-Creating items from dicts:+ Creating items from dicts:->>> Product({'name': 'Laptop PC', 'price': 1500})-Product(price=1500, name='Laptop PC')+ >>> Product({"name": "Laptop PC", "price": 1500})+ Product(price=1500, name='Laptop PC')->>> Product({'name': 'Laptop PC', 'lala': 1500}) # warning: unknown field in dict-Traceback (most recent call last):- ...-KeyError: 'Product does not support field: lala'+ >>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict+ Traceback (most recent call last):+ ...+ KeyError: 'Product does not support field: lala'Extending Item subclassesdiff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rstindex 33441838a..cd8914644 100644--- a/docs/topics/leaks.rst+++ b/docs/topics/leaks.rst@@ -70,13 +70,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function::telnet localhost 6023- >>> prefs()- Live References+ .. code-block:: pycon- ExampleSpider 1 oldest: 15s ago- HtmlResponse 10 oldest: 1s ago- Selector 2 oldest: 0s ago- FormRequest 878 oldest: 7s ago+ >>> prefs()+ Live References++ ExampleSpider 1 oldest: 15s ago+ HtmlResponse 10 oldest: 1s ago+ Selector 2 oldest: 0s ago+ FormRequest 878 oldest: 7s agoAs you can see, that report also shows the "age" of the oldest object in eachclass. If you're running multiple spiders per process chances are you can@@ -114,7 +116,9 @@ a priori, of course) by using the ``trackref`` tool.After the crawler is running for a few minutes and we notice its memory usagehas grown a lot, we can enter its telnet console and check the live-references::+references:++.. code-block:: pycon>>> prefs()Live References@@ -134,19 +138,23 @@ generating the leaks (passing response references inside requests).Sometimes extra information about live objects can be helpful.Let's check the oldest response:->>> from scrapy.utils.trackref import get_oldest->>> r = get_oldest('HtmlResponse')->>> r.url-'http://www.somenastyspider.com/product.php?pid=123'+.. code-block:: pycon++ >>> from scrapy.utils.trackref import get_oldest+ >>> r = get_oldest("HtmlResponse")+ >>> r.url+ 'http://www.somenastyspider.com/product.php?pid=123'If you want to iterate over all objects, instead of getting the oldest one, youcan use the :func:`scrapy.utils.trackref.iter_all` function:->>> from scrapy.utils.trackref import iter_all->>> [r.url for r in iter_all('HtmlResponse')]-['http://www.somenastyspider.com/product.php?pid=123',- 'http://www.somenastyspider.com/product.php?pid=584',-...]+.. code-block:: pycon++ >>> from scrapy.utils.trackref import iter_all+ >>> [r.url for r in iter_all("HtmlResponse")]+ ['http://www.somenastyspider.com/product.php?pid=123',+ 'http://www.somenastyspider.com/product.php?pid=584',+ ...]Too many spiders?-----------------@@ -157,8 +165,10 @@ For this reason, that function has a ``ignore`` argument which can be used toignore a particular class (and all its subclasses). Forexample, this won't show any live references to spiders:->>> from scrapy.spiders import Spider->>> prefs(ignore=Spider)+.. code-block:: pycon++ >>> from scrapy.spiders import Spider+ >>> prefs(ignore=Spider).. module:: scrapy.utils.trackref:synopsis: Track references of live objects@@ -216,30 +226,32 @@ If you use ``pip``, you can install muppy with the following command::Here's an example to view all Python objects available inthe heap using muppy:->>> from pympler import muppy->>> all_objects = muppy.get_objects()->>> len(all_objects)-28667->>> from pympler import summary->>> suml = summary.summarize(all_objects)->>> summary.print_(suml)- types | # objects | total size-==================================== | =========== | ============- <class 'str | 9822 | 1.10 MB- <class 'dict | 1658 | 856.62 KB- <class 'type | 436 | 443.60 KB- <class 'code | 2974 | 419.56 KB- <class '_io.BufferedWriter | 2 | 256.34 KB- <class 'set | 420 | 159.88 KB- <class '_io.BufferedReader | 1 | 128.17 KB- <class 'wrapper_descriptor | 1130 | 88.28 KB- <class 'tuple | 1304 | 86.57 KB- <class 'weakref | 1013 | 79.14 KB- <class 'builtin_function_or_method | 958 | 67.36 KB- <class 'method_descriptor | 865 | 60.82 KB- <class 'abc.ABCMeta | 62 | 59.96 KB- <class 'list | 446 | 58.52 KB- <class 'int | 1425 | 43.20 KB+.. code-block:: pycon++ >>> from pympler import muppy+ >>> all_objects = muppy.get_objects()+ >>> len(all_objects)+ 28667+ >>> from pympler import summary+ >>> suml = summary.summarize(all_objects)+ >>> summary.print_(suml)+ types | # objects | total size+ ==================================== | =========== | ============+ <class 'str | 9822 | 1.10 MB+ <class 'dict | 1658 | 856.62 KB+ <class 'type | 436 | 443.60 KB+ <class 'code | 2974 | 419.56 KB+ <class '_io.BufferedWriter | 2 | 256.34 KB+ <class 'set | 420 | 159.88 KB+ <class '_io.BufferedReader | 1 | 128.17 KB+ <class 'wrapper_descriptor | 1130 | 88.28 KB+ <class 'tuple | 1304 | 86.57 KB+ <class 'weakref | 1013 | 79.14 KB+ <class 'builtin_function_or_method | 958 | 67.36 KB+ <class 'method_descriptor | 865 | 60.82 KB+ <class 'abc.ABCMeta | 62 | 59.96 KB+ <class 'list | 446 | 58.52 KB+ <class 'int | 1425 | 43.20 KBFor more info about muppy, refer to the `muppy documentation`_.diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rstindex eaad926d1..45e19ef07 100644--- a/docs/topics/loaders.rst+++ b/docs/topics/loaders.rst@@ -250,12 +250,15 @@ metadata. Here is an example:output_processor=TakeFirst(),)->>> from scrapy.loader import ItemLoader->>> il = ItemLoader(item=Product())->>> il.add_value('name', ['Welcome to my', '<strong>website</strong>'])->>> il.add_value('price', ['€', '<span>1000</span>'])->>> il.load_item()-{'name': 'Welcome to my website', 'price': '1000'}++.. code-block:: pycon++ >>> from scrapy.loader import ItemLoader+ >>> il = ItemLoader(item=Product())+ >>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])+ >>> il.add_value("price", ["€", "<span>1000</span>"])+ >>> il.load_item()+ {'name': 'Welcome to my website', 'price': '1000'}The precedence order, for both input and output processors, is as follows:diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 090554e6e..fd38eb504 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -1243,8 +1243,10 @@ TextResponse objects``str(response.body)`` is not a correct way to convert the responsebody into a string:- >>> str(b'body')- "b'body'"+ .. code-block:: pycon++ >>> str(b"body")+ "b'body'".. attribute:: TextResponse.encodingdiff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rstindex ede3ca7cd..80a979650 100644--- a/docs/topics/selectors.rst+++ b/docs/topics/selectors.rst@@ -51,16 +51,20 @@ Constructing selectorsResponse objects expose a :class:`~scrapy.Selector` instanceon ``.selector`` attribute:->>> response.selector.xpath('//span/text()').get()-'good'+.. code-block:: pycon++ >>> response.selector.xpath("//span/text()").get()+ 'good'Querying responses using XPath and CSS is so common that responses include twomore shortcuts: ``response.xpath()`` and ``response.css()``:->>> response.xpath('//span/text()').get()-'good'->>> response.css('span::text').get()-'good'+.. code-block:: pycon++ >>> response.xpath("//span/text()").get()+ 'good'+ >>> response.css("span::text").get()+ 'good'Scrapy selectors are instances of :class:`~scrapy.Selector` classconstructed by passing either :class:`~scrapy.http.TextResponse` object or@@ -75,19 +79,23 @@ you can also ensure the response body is parsed only once.But if required, it is possible to use ``Selector`` directly.Constructing from text:->>> from scrapy.selector import Selector->>> body = '<html><body><span>good</span></body></html>'->>> Selector(text=body).xpath('//span/text()').get()-'good'+.. code-block:: pycon++ >>> from scrapy.selector import Selector+ >>> body = "<html><body><span>good</span></body></html>"+ >>> Selector(text=body).xpath("//span/text()").get()+ 'good'Constructing from response - :class:`~scrapy.http.HtmlResponse` is one of:class:`~scrapy.http.TextResponse` subclasses:->>> from scrapy.selector import Selector->>> from scrapy.http import HtmlResponse->>> response = HtmlResponse(url='http://example.com', body=body)->>> Selector(response=response).xpath('//span/text()').get()-'good'+.. code-block:: pycon++ >>> from scrapy.selector import Selector+ >>> from scrapy.http import HtmlResponse+ >>> response = HtmlResponse(url="http://example.com", body=body)+ >>> Selector(response=response).xpath("//span/text()").get()+ 'good'``Selector`` automatically chooses the best parsing rules(XML vs HTML) based on input type.@@ -124,16 +132,20 @@ Since we're dealing with HTML, the selector will automatically use an HTML parseSo, by looking at the :ref:`HTML code <topics-selectors-htmlcode>` of thatpage, let's construct an XPath for selecting the text inside the title tag:->>> response.xpath('//title/text()')-[<Selector xpath='//title/text()' data='Example website'>]+.. code-block:: pycon++ >>> response.xpath("//title/text()")+ [<Selector xpath='//title/text()' data='Example website'>]To actually extract the textual data, you must call the selector ``.get()``or ``.getall()`` methods, as follows:->>> response.xpath('//title/text()').getall()-['Example website']->>> response.xpath('//title/text()').get()-'Example website'+.. code-block:: pycon++ >>> response.xpath("//title/text()").getall()+ ['Example website']+ >>> response.xpath("//title/text()").get()+ 'Example website'``.get()`` always returns a single result; if there are several matches,content of a first match is returned; if there are no matches, None@@ -142,98 +154,116 @@ is returned. ``.getall()`` returns a list with all results.Notice that CSS selectors can select text or attribute nodes using CSS3pseudo-elements:->>> response.css('title::text').get()-'Example website'+.. code-block:: pycon++ >>> response.css("title::text").get()+ 'Example website'As you can see, ``.xpath()`` and ``.css()`` methods return a:class:`~scrapy.selector.SelectorList` instance, which is a list of newselectors. This API can be used for quickly selecting nested data:->>> response.css('img').xpath('@src').getall()-['image1_thumb.jpg',- 'image2_thumb.jpg',- 'image3_thumb.jpg',- 'image4_thumb.jpg',- 'image5_thumb.jpg']+.. code-block:: pycon++ >>> response.css("img").xpath("@src").getall()+ ['image1_thumb.jpg',+ 'image2_thumb.jpg',+ 'image3_thumb.jpg',+ 'image4_thumb.jpg',+ 'image5_thumb.jpg']If you want to extract only the first matched element, you can call theselector ``.get()`` (or its alias ``.extract_first()`` commonly used inprevious Scrapy versions):->>> response.xpath('//div[@id="images"]/a/text()').get()-'Name: My image 1 '+.. code-block:: pycon++ >>> response.xpath('//div[@id="images"]/a/text()').get()+ 'Name: My image 1 'It returns ``None`` if no element was found:->>> response.xpath('//div[@id="not-exists"]/text()').get() is None-True+.. code-block:: pycon++ >>> response.xpath('//div[@id="not-exists"]/text()').get() is None+ TrueA default return value can be provided as an argument, to be used insteadof ``None``:->>> response.xpath('//div[@id="not-exists"]/text()').get(default='not-found')-'not-found'+.. code-block:: pycon++ >>> response.xpath('//div[@id="not-exists"]/text()').get(default="not-found")+ 'not-found'Instead of using e.g. ``'@src'`` XPath it is possible to query for attributesusing ``.attrib`` property of a :class:`~scrapy.Selector`:->>> [img.attrib['src'] for img in response.css('img')]-['image1_thumb.jpg',- 'image2_thumb.jpg',- 'image3_thumb.jpg',- 'image4_thumb.jpg',- 'image5_thumb.jpg']+.. code-block:: pycon++ >>> [img.attrib["src"] for img in response.css("img")]+ ['image1_thumb.jpg',+ 'image2_thumb.jpg',+ 'image3_thumb.jpg',+ 'image4_thumb.jpg',+ 'image5_thumb.jpg']As a shortcut, ``.attrib`` is also available on SelectorList directly;it returns attributes for the first matching element:->>> response.css('img').attrib['src']-'image1_thumb.jpg'+.. code-block:: pycon++ >>> response.css("img").attrib["src"]+ 'image1_thumb.jpg'This is most useful when only a single result is expected, e.g. when selectingby id, or selecting unique elements on a web page:->>> response.css('base').attrib['href']-'http://example.com/'+.. code-block:: pycon++ >>> response.css("base").attrib["href"]+ 'http://example.com/'Now we're going to get the base URL and some image links:->>> response.xpath('//base/@href').get()-'http://example.com/'-->>> response.css('base::attr(href)').get()-'http://example.com/'-->>> response.css('base').attrib['href']-'http://example.com/'-->>> response.xpath('//a[contains(@href, "image")]/@href').getall()-['image1.html',- 'image2.html',- 'image3.html',- 'image4.html',- 'image5.html']-->>> response.css('a[href*=image]::attr(href)').getall()-['image1.html',- 'image2.html',- 'image3.html',- 'image4.html',- 'image5.html']-->>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()-['image1_thumb.jpg',- 'image2_thumb.jpg',- 'image3_thumb.jpg',- 'image4_thumb.jpg',- 'image5_thumb.jpg']-->>> response.css('a[href*=image] img::attr(src)').getall()-['image1_thumb.jpg',- 'image2_thumb.jpg',- 'image3_thumb.jpg',- 'image4_thumb.jpg',- 'image5_thumb.jpg']+.. code-block:: pycon++ >>> response.xpath("//base/@href").get()+ 'http://example.com/'++ >>> response.css("base::attr(href)").get()+ 'http://example.com/'++ >>> response.css("base").attrib["href"]+ 'http://example.com/'++ >>> response.xpath('//a[contains(@href, "image")]/@href').getall()+ ['image1.html',+ 'image2.html',+ 'image3.html',+ 'image4.html',+ 'image5.html']++ >>> response.css("a[href*=image]::attr(href)").getall()+ ['image1.html',+ 'image2.html',+ 'image3.html',+ 'image4.html',+ 'image5.html']++ >>> response.xpath('//a[contains(@href, "image")]/img/@src').getall()+ ['image1_thumb.jpg',+ 'image2_thumb.jpg',+ 'image3_thumb.jpg',+ 'image4_thumb.jpg',+ 'image5_thumb.jpg']++ >>> response.css("a[href*=image] img::attr(src)").getall()+ ['image1_thumb.jpg',+ 'image2_thumb.jpg',+ 'image3_thumb.jpg',+ 'image4_thumb.jpg',+ 'image5_thumb.jpg'].. _topics-selectors-css-extensions:@@ -260,45 +290,55 @@ Examples:* ``title::text`` selects children text nodes of a descendant ``<title>`` element:->>> response.css('title::text').get()-'Example website'+.. code-block:: pycon++ >>> response.css("title::text").get()+ 'Example website'* ``*::text`` selects all descendant text nodes of the current selector context:->>> response.css('#images *::text').getall()-['\n ',- 'Name: My image 1 ',- '\n ',- 'Name: My image 2 ',- '\n ',- 'Name: My image 3 ',- '\n ',- 'Name: My image 4 ',- '\n ',- 'Name: My image 5 ',- '\n ']+.. code-block:: pycon++ >>> response.css("#images *::text").getall()+ ['\n ',+ 'Name: My image 1 ',+ '\n ',+ 'Name: My image 2 ',+ '\n ',+ 'Name: My image 3 ',+ '\n ',+ 'Name: My image 4 ',+ '\n ',+ 'Name: My image 5 ',+ '\n ']* ``foo::text`` returns no results if ``foo`` element exists, but containsno text (i.e. text is empty):->>> response.css('img::text').getall()-[]+.. code-block:: pycon++ >>> response.css("img::text").getall()+ []++ is means ``.css('foo::text').get()`` could return None even if an element+ ists. Use ``default=''`` if you always want a string:- This means ``.css('foo::text').get()`` could return None even if an element- exists. Use ``default=''`` if you always want a string:+.. code-block:: pycon->>> response.css('img::text').get()->>> response.css('img::text').get(default='')-''+ >>> response.css("img::text").get()+ >>> response.css("img::text").get(default="")+ ''* ``a::attr(href)`` selects the *href* attribute value of descendant links:->>> response.css('a::attr(href)').getall()-['image1.html',- 'image2.html',- 'image3.html',- 'image4.html',- 'image5.html']+.. code-block:: pycon++ >>> response.css("a::attr(href)").getall()+ ['image1.html',+ 'image2.html',+ 'image3.html',+ 'image4.html',+ 'image5.html'].. note::See also: :ref:`selecting-attributes`.@@ -319,23 +359,26 @@ The selection methods (``.xpath()`` or ``.css()``) return a list of selectorsof the same type, so you can call the selection methods for those selectorstoo. Here's an example:->>> links = response.xpath('//a[contains(@href, "image")]')->>> links.getall()-['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',- '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',- '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',- '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',- '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']-->>> for index, link in enumerate(links):-... href_xpath = link.xpath('@href').get()-... img_xpath = link.xpath('img/@src').get()-... print(f'Link number {index} points to url {href_xpath!r} and image {img_xpath!r}')-Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'-Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'-Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'-Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'-Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'+.. code-block:: pycon++ >>> links = response.xpath('//a[contains(@href, "image")]')+ >>> links.getall()+ ['<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',+ '<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',+ '<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',+ '<a href="image4.html">Name: My image 4 <br><img src="image4_thumb.jpg"></a>',+ '<a href="image5.html">Name: My image 5 <br><img src="image5_thumb.jpg"></a>']++ >>> for index, link in enumerate(links):+ ... href_xpath = link.xpath("@href").get()+ ... img_xpath = link.xpath("img/@src").get()+ ... print(f"Link number {index} points to url {href_xpath!r} and image {img_xpath!r}")+ ...+ Link number 0 points to url 'image1.html' and image 'image1_thumb.jpg'+ Link number 1 points to url 'image2.html' and image 'image2_thumb.jpg'+ Link number 2 points to url 'image3.html' and image 'image3_thumb.jpg'+ Link number 3 points to url 'image4.html' and image 'image4_thumb.jpg'+ Link number 4 points to url 'image5.html' and image 'image5_thumb.jpg'.. _selecting-attributes:@@ -345,8 +388,10 @@ Selecting element attributesThere are several ways to get a value of an attribute. First, one can useXPath syntax:->>> response.xpath("//a/@href").getall()-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']+.. code-block:: pycon++ >>> response.xpath("//a/@href").getall()+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']XPath syntax has a few advantages: it is a standard XPath feature, and``@attributes`` can be used in other parts of an XPath expression - e.g.@@ -355,30 +400,38 @@ it is possible to filter by attribute value.Scrapy also provides an extension to CSS selectors (``::attr(...)``)which allows to get attribute values:->>> response.css('a::attr(href)').getall()-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']+.. code-block:: pycon++ >>> response.css("a::attr(href)").getall()+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']In addition to that, there is a ``.attrib`` property of Selector.You can use it if you prefer to lookup attributes in Pythoncode, without using XPaths or CSS extensions:->>> [a.attrib['href'] for a in response.css('a')]-['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']+.. code-block:: pycon++ >>> [a.attrib["href"] for a in response.css("a")]+ ['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']This property is also available on SelectorList; it returns a dictionarywith attributes of a first matching element. It is convenient to use whena selector is expected to give a single result (e.g. when selecting by elementID, or when selecting an unique element on a page):->>> response.css('base').attrib-{'href': 'http://example.com/'}->>> response.css('base').attrib['href']-'http://example.com/'+.. code-block:: pycon++ >>> response.css("base").attrib+ {'href': 'http://example.com/'}+ >>> response.css("base").attrib["href"]+ 'http://example.com/'``.attrib`` property of an empty SelectorList is empty:->>> response.css('foo').attrib-{}+.. code-block:: pycon++ >>> response.css("foo").attrib+ {}Using selectors with regular expressions----------------------------------------@@ -391,19 +444,23 @@ can't construct nested ``.re()`` calls.Here's an example used to extract image names from the :ref:`HTML code<topics-selectors-htmlcode>` above:->>> response.xpath('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')-['My image 1',- 'My image 2',- 'My image 3',- 'My image 4',- 'My image 5']+.. code-block:: pycon++ >>> response.xpath('//a[contains(@href, "image")]/text()').re(r"Name:\s*(.*)")+ ['My image 1',+ 'My image 2',+ 'My image 3',+ 'My image 4',+ 'My image 5']There's an additional helper reciprocating ``.get()`` (and itsalias ``.extract_first()``) for ``.re()``, named ``.re_first()``.Use it to extract just the first matching string:->>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r'Name:\s*(.*)')-'My image 1'+.. code-block:: pycon++ >>> response.xpath('//a[contains(@href, "image")]/text()').re_first(r"Name:\s*(.*)")+ 'My image 1'.. _old-extraction-api:@@ -423,28 +480,36 @@ The following examples show how these methods map to each other.1. ``SelectorList.get()`` is the same as ``SelectorList.extract_first()``:- >>> response.css('a::attr(href)').get()+.. code-block:: pycon++ >>> response.css("a::attr(href)").get()'image1.html'- >>> response.css('a::attr(href)').extract_first()+ >>> response.css("a::attr(href)").extract_first()'image1.html'2. ``SelectorList.getall()`` is the same as ``SelectorList.extract()``:- >>> response.css('a::attr(href)').getall()+.. code-block:: pycon++ >>> response.css("a::attr(href)").getall()['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']- >>> response.css('a::attr(href)').extract()+ >>> response.css("a::attr(href)").extract()['image1.html', 'image2.html', 'image3.html', 'image4.html', 'image5.html']3. ``Selector.get()`` is the same as ``Selector.extract()``:- >>> response.css('a::attr(href)')[0].get()+.. code-block:: pycon++ >>> response.css("a::attr(href)")[0].get()'image1.html'- >>> response.css('a::attr(href)')[0].extract()+ >>> response.css("a::attr(href)")[0].extract()'image1.html'4. For consistency, there is also ``Selector.getall()``, which returns a list:- >>> response.css('a::attr(href)')[0].getall()+.. code-block:: pycon++ >>> response.css("a::attr(href)")[0].getall()['image1.html']So, the main difference is that output of ``.get()`` and ``.getall()`` methods@@ -482,24 +547,34 @@ with ``/``, that XPath will be absolute to the document and not relative to theFor example, suppose you want to extract all ``<p>`` elements inside ``<div>``elements. First, you would get all ``<div>`` elements:->>> divs = response.xpath('//div')+.. code-block:: pycon+ >>> divs = response.xpath("//div")At first, you may be tempted to use the following approach, which is wrong, asit actually extracts all ``<p>`` elements from the document, not only thoseinside ``<div>`` elements:->>> for p in divs.xpath('//p'): # this is wrong - gets all <p> from the whole document-... print(p.get())+.. code-block:: pycon++ >>> for p in divs.xpath("//p"): # this is wrong - gets all <p> from the whole document+ ... print(p.get())+ ...This is the proper way to do it (note the dot prefixing the ``.//p`` XPath):->>> for p in divs.xpath('.//p'): # extracts all <p> inside-... print(p.get())+.. code-block:: pycon++ >>> for p in divs.xpath(".//p"): # extracts all <p> inside+ ... print(p.get())+ ...Another common case would be to extract all direct ``<p>`` children:->>> for p in divs.xpath('p'):-... print(p.get())+.. code-block:: pycon++ >>> for p in divs.xpath("p"):+ ... print(p.get())+ ...For more details about relative XPaths see the `Location Paths`_ section in theXPath specification.@@ -522,10 +597,14 @@ class name that shares the string ``someclass``.As it turns out, Scrapy selectors allow you to chain selectors, so most of the timeyou can just select by class using CSS and then switch to XPath when needed:->>> from scrapy import Selector->>> sel = Selector(text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>')->>> sel.css('.shout').xpath('./time/@datetime').getall()-['2014-07-23 19:00']+.. code-block:: pycon++ >>> from scrapy import Selector+ >>> sel = Selector(+ ... text='<div class="hero shout"><time datetime="2014-07-23 19:00">Special date</time></div>'+ ... )+ >>> sel.css(".shout").xpath("./time/@datetime").getall()+ ['2014-07-23 19:00']This is cleaner than using the verbose XPath trick shown above. Just rememberto use the ``.`` in the XPath expressions that will follow.@@ -539,39 +618,51 @@ Beware of the difference between //node[1] and (//node)[1]Example:->>> from scrapy import Selector->>> sel = Selector(text="""-....: <ul class="list">-....: <li>1</li>-....: <li>2</li>-....: <li>3</li>-....: </ul>-....: <ul class="list">-....: <li>4</li>-....: <li>5</li>-....: <li>6</li>-....: </ul>""")->>> xp = lambda x: sel.xpath(x).getall()+.. code-block:: pycon++ >>> from scrapy import Selector+ >>> sel = Selector(+ ... text="""+ ... <ul class="list">+ ... <li>1</li>+ ... <li>2</li>+ ... <li>3</li>+ ... </ul>+ ... <ul class="list">+ ... <li>4</li>+ ... <li>5</li>+ ... <li>6</li>+ ... </ul>"""+ ... )+ >>> xp = lambda x: sel.xpath(x).getall()This gets all first ``<li>`` elements under whatever it is its parent:->>> xp("//li[1]")-['<li>1</li>', '<li>4</li>']+.. code-block:: pycon++ >>> xp("//li[1]")+ ['<li>1</li>', '<li>4</li>']And this gets the first ``<li>`` element in the whole document:->>> xp("(//li)[1]")-['<li>1</li>']+.. code-block:: pycon++ >>> xp("(//li)[1]")+ ['<li>1</li>']This gets all first ``<li>`` elements under an ``<ul>`` parent:->>> xp("//ul/li[1]")-['<li>1</li>', '<li>4</li>']+.. code-block:: pycon++ >>> xp("//ul/li[1]")+ ['<li>1</li>', '<li>4</li>']And this gets the first ``<li>`` element under an ``<ul>`` parent in the whole document:->>> xp("(//ul/li)[1]")-['<li>1</li>']+.. code-block:: pycon++ >>> xp("(//ul/li)[1]")+ ['<li>1</li>']Using text nodes in a condition-------------------------------@@ -585,32 +676,44 @@ a string function like ``contains()`` or ``starts-with()``, it results in the teExample:->>> from scrapy import Selector->>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')+.. code-block:: pycon++ >>> from scrapy import Selector+ >>> sel = Selector(+ ... text='<a href="#">Click here to go to the <strong>Next Page</strong></a>'+ ... )Converting a *node-set* to string:->>> sel.xpath('//a//text()').getall() # take a peek at the node-set-['Click here to go to the ', 'Next Page']->>> sel.xpath("string(//a[1]//text())").getall() # convert it to string-['Click here to go to the ']+.. code-block:: pycon++ >>> sel.xpath("//a//text()").getall() # take a peek at the node-set+ ['Click here to go to the ', 'Next Page']+ >>> sel.xpath("string(//a[1]//text())").getall() # convert it to string+ ['Click here to go to the ']A *node* converted to a string, however, puts together the text of itself plus of all its descendants:->>> sel.xpath("//a[1]").getall() # select the first node-['<a href="#">Click here to go to the <strong>Next Page</strong></a>']->>> sel.xpath("string(//a[1])").getall() # convert it to string-['Click here to go to the Next Page']+.. code-block:: pycon++ >>> sel.xpath("//a[1]").getall() # select the first node+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>']+ >>> sel.xpath("string(//a[1])").getall() # convert it to string+ ['Click here to go to the Next Page']So, using the ``.//text()`` node-set won't select anything in this case:->>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()-[]+.. code-block:: pycon++ >>> sel.xpath("//a[contains(.//text(), 'Next Page')]").getall()+ []But using the ``.`` to mean the node, works:->>> sel.xpath("//a[contains(., 'Next Page')]").getall()-['<a href="#">Click here to go to the <strong>Next Page</strong></a>']+.. code-block:: pycon++ >>> sel.xpath("//a[contains(., 'Next Page')]").getall()+ ['<a href="#">Click here to go to the <strong>Next Page</strong></a>'].. _`XPath string function`: https://www.w3.org/TR/xpath/all/#section-String-Functions@@ -628,15 +731,19 @@ which are then substituted with values passed with the query.Here's an example to match an element based on its "id" attribute value,without hard-coding it (that was shown previously):->>> # `$val` used in the expression, a `val` argument needs to be passed->>> response.xpath('//div[@id=$val]/a/text()', val='images').get()-'Name: My image 1 '+.. code-block:: pycon++ >>> # `$val` used in the expression, a `val` argument needs to be passed+ >>> response.xpath("//div[@id=$val]/a/text()", val="images").get()+ 'Name: My image 1 'Here's another example, to find the "id" attribute of a ``<div>`` tag containingfive ``<a>`` children (here we pass the value ``5`` as an integer):->>> response.xpath('//div[count(a)=$cnt]/@id', cnt=5).get()-'images'+.. code-block:: pycon++ >>> response.xpath("//div[count(a)=$cnt]/@id", cnt=5).get()+ 'images'All variable references must have a binding value when calling ``.xpath()``(otherwise you'll get a ``ValueError: XPath error:`` exception).@@ -688,17 +795,21 @@ You can see several namespace declarations including a defaultOnce in the shell we can try selecting all ``<link>`` objects and see that itdoesn't work (because the Atom XML namespace is obfuscating those nodes):->>> response.xpath("//link")-[]+.. code-block:: pycon++ >>> response.xpath("//link")+ []But once we call the :meth:`Selector.remove_namespaces` method, allnodes can be accessed directly by their names:->>> response.selector.remove_namespaces()->>> response.xpath("//link")-[<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,- <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,- ...+.. code-block:: pycon++ >>> response.selector.remove_namespaces()+ >>> response.xpath("//link")+ [<Selector xpath='//link' data='<link rel="alternate" type="text/html" h'>,+ <Selector xpath='//link' data='<link rel="next" type="application/atom+'>,+ ...If you wonder why the namespace removal procedure isn't always called by defaultinstead of having to call it manually, this is because of two reasons, which, in order@@ -735,23 +846,25 @@ The ``test()`` function, for example, can prove quite useful when XPath'sExample selecting links in list item with a "class" attribute ending with a digit:->>> from scrapy import Selector->>> doc = """-... <div>-... <ul>-... <li class="item-0"><a href="link1.html">first item</a></li>-... <li class="item-1"><a href="link2.html">second item</a></li>-... <li class="item-inactive"><a href="link3.html">third item</a></li>-... <li class="item-1"><a href="link4.html">fourth item</a></li>-... <li class="item-0"><a href="link5.html">fifth item</a></li>-... </ul>-... </div>-... """->>> sel = Selector(text=doc, type="html")->>> sel.xpath('//li//@href').getall()-['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']->>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()-['link1.html', 'link2.html', 'link4.html', 'link5.html']+.. code-block:: pycon++ >>> from scrapy import Selector+ >>> doc = """+ ... <div>+ ... <ul>+ ... <li class="item-0"><a href="link1.html">first item</a></li>+ ... <li class="item-1"><a href="link2.html">second item</a></li>+ ... <li class="item-inactive"><a href="link3.html">third item</a></li>+ ... <li class="item-1"><a href="link4.html">fourth item</a></li>+ ... <li class="item-0"><a href="link5.html">fifth item</a></li>+ ... </ul>+ ... </div>+ ... """+ >>> sel = Selector(text=doc, type="html")+ >>> sel.xpath("//li//@href").getall()+ ['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']+ >>> sel.xpath('//li[re:test(@class, "item-\d$")]//@href').getall()+ ['link1.html', 'link2.html', 'link4.html', 'link5.html'].. warning:: C library ``libxslt`` doesn't natively support EXSLT regularexpressions so `lxml`_'s implementation uses hooks to Python's ``re`` module.@@ -765,7 +878,9 @@ These can be handy for excluding parts of a document tree beforeextracting text elements for example.Example extracting microdata (sample content taken from https://schema.org/Product)-with groups of itemscopes and corresponding itemprops::+with groups of itemscopes and corresponding itemprops:++.. code-block:: pycon>>> doc = """... <div itemscope itemtype="http://schema.org/Product">@@ -776,19 +891,15 @@ with groups of itemscopes and corresponding itemprops::... Rated <span itemprop="ratingValue">3.5</span>/5... based on <span itemprop="reviewCount">11</span> customer reviews... </div>- ...... <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">... <span itemprop="price">$55.00</span>... <link itemprop="availability" href="http://schema.org/InStock" />In stock... </div>- ...... Product description:... <span itemprop="description">0.7 cubic feet countertop microwave.... Has six preset cooking categories and convenience features like... Add-A-Minute and Child Lock.</span>- ...... Customer reviews:- ...... <div itemprop="review" itemscope itemtype="http://schema.org/Review">... <span itemprop="name">Not a happy camper</span> -... by <span itemprop="author">Ellie</span>,@@ -801,7 +912,6 @@ with groups of itemscopes and corresponding itemprops::... <span itemprop="description">The lamp burned out and now I have to replace... it. </span>... </div>- ...... <div itemprop="review" itemscope itemtype="http://schema.org/Review">... <span itemprop="name">Value purchase</span> -... by <span itemprop="author">Lucas</span>,@@ -818,13 +928,16 @@ with groups of itemscopes and corresponding itemprops::... </div>... """>>> sel = Selector(text=doc, type="html")- >>> for scope in sel.xpath('//div[@itemscope]'):- ... print("current scope:", scope.xpath('@itemtype').getall())- ... props = scope.xpath('''+ >>> for scope in sel.xpath("//div[@itemscope]"):+ ... print("current scope:", scope.xpath("@itemtype").getall())+ ... props = scope.xpath(+ ... """... set:difference(./descendant::*/@itemprop,- ... .//*[@itemscope]/*/@itemprop)''')+ ... .//*[@itemscope]/*/@itemprop)"""+ ... )... print(f" properties: {props.getall()}")... print("")+ ...current scope: ['http://schema.org/Product']properties: ['name', 'aggregateRating', 'offers', 'description', 'review', 'review']@@ -876,13 +989,15 @@ For the following HTML::You can use it like this:->>> response.xpath('//p[has-class("foo")]')-[<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,- <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]->>> response.xpath('//p[has-class("foo", "bar-baz")]')-[<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]->>> response.xpath('//p[has-class("foo", "bar")]')-[]+.. code-block:: pycon++ >>> response.xpath('//p[has-class("foo")]')+ [<Selector xpath='//p[has-class("foo")]' data='<p class="foo bar-baz">First</p>'>,+ <Selector xpath='//p[has-class("foo")]' data='<p class="foo">Second</p>'>]+ >>> response.xpath('//p[has-class("foo", "bar-baz")]')+ [<Selector xpath='//p[has-class("foo", "bar-baz")]' data='<p class="foo bar-baz">First</p>'>]+ >>> response.xpath('//p[has-class("foo", "bar")]')+ []So XPath ``//p[has-class("foo", "bar-baz")]`` is roughly equivalent to CSS``p.foo.bar-baz``. Please note, that it is slower in most of the cases,diff --git a/docs/topics/shell.rst b/docs/topics/shell.rstindex 69a90f620..4898843e4 100644--- a/docs/topics/shell.rst+++ b/docs/topics/shell.rst@@ -191,44 +191,46 @@ all start with the ``[s]`` prefix)::After that, we can start playing with the objects:->>> response.xpath('//title/text()').get()-'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'-->>> fetch("https://old.reddit.com/")-->>> response.xpath('//title/text()').get()-'reddit: the front page of the internet'-->>> request = request.replace(method="POST")-->>> fetch(request)-->>> response.status-404-->>> from pprint import pprint-->>> pprint(response.headers)-{'Accept-Ranges': ['bytes'],- 'Cache-Control': ['max-age=0, must-revalidate'],- 'Content-Type': ['text/html; charset=UTF-8'],- 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],- 'Server': ['snooserv'],- 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',- 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',- 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',- 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],- 'Vary': ['accept-encoding'],- 'Via': ['1.1 varnish'],- 'X-Cache': ['MISS'],- 'X-Cache-Hits': ['0'],- 'X-Content-Type-Options': ['nosniff'],- 'X-Frame-Options': ['SAMEORIGIN'],- 'X-Moose': ['majestic'],- 'X-Served-By': ['cache-cdg8730-CDG'],- 'X-Timer': ['S1481214079.394283,VS0,VE159'],- 'X-Ua-Compatible': ['IE=edge'],- 'X-Xss-Protection': ['1; mode=block']}+.. code-block:: pycon++ >>> response.xpath("//title/text()").get()+ 'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'++ >>> fetch("https://old.reddit.com/")++ >>> response.xpath("//title/text()").get()+ 'reddit: the front page of the internet'++ >>> request = request.replace(method="POST")++ >>> fetch(request)++ >>> response.status+ 404++ >>> from pprint import pprint++ >>> pprint(response.headers)+ {'Accept-Ranges': ['bytes'],+ 'Cache-Control': ['max-age=0, must-revalidate'],+ 'Content-Type': ['text/html; charset=UTF-8'],+ 'Date': ['Thu, 08 Dec 2016 16:21:19 GMT'],+ 'Server': ['snooserv'],+ 'Set-Cookie': ['loid=KqNLou0V9SKMX4qb4n; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',+ 'loidcreated=2016-12-08T16%3A21%3A19.445Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',+ 'loid=vi0ZVe4NkxNWdlH7r7; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure',+ 'loidcreated=2016-12-08T16%3A21%3A19.459Z; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Sat, 08-Dec-2018 16:21:19 GMT; secure'],+ 'Vary': ['accept-encoding'],+ 'Via': ['1.1 varnish'],+ 'X-Cache': ['MISS'],+ 'X-Cache-Hits': ['0'],+ 'X-Content-Type-Options': ['nosniff'],+ 'X-Frame-Options': ['SAMEORIGIN'],+ 'X-Moose': ['majestic'],+ 'X-Served-By': ['cache-cdg8730-CDG'],+ 'X-Timer': ['S1481214079.394283,VS0,VE159'],+ 'X-Ua-Compatible': ['IE=edge'],+ 'X-Xss-Protection': ['1; mode=block']}.. _topics-shell-inspect-response:@@ -279,14 +281,18 @@ When you run the spider, you will get something similar to this::Then, you can check if the extraction code is working:->>> response.xpath('//h1[@class="fn"]')-[]+.. code-block:: pycon++ >>> response.xpath('//h1[@class="fn"]')+ []Nope, it doesn't. So you can open the response in your web browser and see ifit's the response you were expecting:->>> view(response)-True+.. code-block:: pycon++ >>> view(response)+ TrueFinally you hit Ctrl-D (or Ctrl-Z in Windows) to exit the shell and resume thecrawling::diff --git a/docs/topics/stats.rst b/docs/topics/stats.rstindex f8b0c4d2a..be8ecb7a5 100644--- a/docs/topics/stats.rst+++ b/docs/topics/stats.rst@@ -68,13 +68,17 @@ Set stat value only if lower than previous:Get stat value:->>> stats.get_value('custom_count')-1+.. code-block:: pycon++ >>> stats.get_value("custom_count")+ 1Get all stats:->>> stats.get_stats()-{'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}+.. code-block:: pycon++ >>> stats.get_stats()+ {'custom_count': 1, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}Available Stats Collectors==========================