Scrapy, a fast high-level web crawling & scraping framework for Python.
DOC .extract_first() -> .get()
Mikhail Korobovca27010d4f8c
diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex 6f1c2c43f..9d7c94d39 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -34,11 +34,11 @@ http://quotes.toscrape.com, following the pagination::def parse(self, response):for quote in response.css('div.quote'):yield {- 'text': quote.css('span.text::text').extract_first(),- 'author': quote.xpath('span/small/text()').extract_first(),+ 'text': quote.css('span.text::text').get(),+ 'author': quote.xpath('span/small/text()').get(),}- next_page = response.css('li.next a::attr("href")').extract_first()+ next_page = response.css('li.next a::attr("href")').get()if next_page is not None:yield response.follow(next_page, self.parse)diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex fa6dc274d..a24cf0f5b 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -269,23 +269,24 @@ The other thing is that the result of calling ``.extract()`` is a list, becausewe're dealing with an instance of :class:`~scrapy.selector.SelectorList`. Whenyou know you just want the first result, as in this case, you can do::- >>> response.css('title::text').extract_first()+ >>> response.css('title::text').get()'Quotes to Scrape'As an alternative, you could've written::- >>> response.css('title::text')[0].extract()+ >>> response.css('title::text')[0].get()'Quotes to Scrape'-However, using ``.extract_first()`` avoids an ``IndexError`` and returns-``None`` when it doesn't find any element matching the selection.+However, using ``.get()`` directly on a :class:`~scrapy.selector.SelectorList`+instance avoids an ``IndexError`` and returns ``None`` when it doesn't+find any element matching the selection.There's a lesson here: for most scraping code, you want it to be resilient toerrors due to things not being found on a page, so that even if some parts failto be scraped, you can at least get **some** data.Besides the :meth:`~scrapy.selector.Selector.extract` and-:meth:`~scrapy.selector.SelectorList.extract_first` methods, you can also use+:meth:`~scrapy.selector.SelectorList.get` methods, you can also usethe :meth:`~scrapy.selector.Selector.re` method to extract using `regularexpressions`::@@ -314,7 +315,7 @@ Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions::>>> response.xpath('//title')[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]- >>> response.xpath('//title/text()').extract_first()+ >>> response.xpath('//title/text()').get()'Quotes to Scrape'XPath expressions are very powerful, and are the foundation of Scrapy@@ -383,10 +384,10 @@ variable, so that we can run our CSS selectors directly on a particular quote::Now, let's extract ``title``, ``author`` and the ``tags`` from that quoteusing the ``quote`` object we just created::- >>> title = quote.css("span.text::text").extract_first()+ >>> title = quote.css("span.text::text").get()>>> title'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'- >>> author = quote.css("small.author::text").extract_first()+ >>> author = quote.css("small.author::text").get()>>> author'Albert Einstein'@@ -401,8 +402,8 @@ Having figured out how to extract each bit, we can now iterate over all thequotes elements and put them together into a Python dictionary::>>> for quote in response.css("div.quote"):- ... text = quote.css("span.text::text").extract_first()- ... author = quote.css("small.author::text").extract_first()+ ... text = quote.css("span.text::text").get()+ ... author = quote.css("small.author::text").get()... tags = quote.css("div.tags a.tag::text").extract()... print(dict(text=text, author=author, tags=tags)){'tags': ['change', 'deep-thoughts', 'thinking', 'world'], 'author': 'Albert Einstein', 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}@@ -434,8 +435,8 @@ in the callback, as you can see below::def parse(self, response):for quote in response.css('div.quote'):yield {- 'text': quote.css('span.text::text').extract_first(),- 'author': quote.css('small.author::text').extract_first(),+ 'text': quote.css('span.text::text').get(),+ 'author': quote.css('small.author::text').get(),'tags': quote.css('div.tags a.tag::text').extract(),}@@ -508,14 +509,14 @@ markup:We can try extracting it in the shell::- >>> response.css('li.next a').extract_first()+ >>> response.css('li.next a').get()'<a href="/page/2/">Next <span aria-hidden="true">→</span></a>'This gets the anchor element, but we want the attribute ``href``. For that,Scrapy supports a CSS extension that let's you select the attribute contents,like this::- >>> response.css('li.next a::attr(href)').extract_first()+ >>> response.css('li.next a::attr(href)').get()'/page/2/'Let's see now our spider modified to recursively follow the link to the next@@ -533,12 +534,12 @@ page, extracting data from it::def parse(self, response):for quote in response.css('div.quote'):yield {- 'text': quote.css('span.text::text').extract_first(),- 'author': quote.css('small.author::text').extract_first(),+ 'text': quote.css('span.text::text').get(),+ 'author': quote.css('small.author::text').get(),'tags': quote.css('div.tags a.tag::text').extract(),}- next_page = response.css('li.next a::attr(href)').extract_first()+ next_page = response.css('li.next a::attr(href)').get()if next_page is not None:next_page = response.urljoin(next_page)yield scrapy.Request(next_page, callback=self.parse)@@ -584,12 +585,12 @@ As a shortcut for creating Request objects you can usedef parse(self, response):for quote in response.css('div.quote'):yield {- 'text': quote.css('span.text::text').extract_first(),- 'author': quote.css('span small::text').extract_first(),+ 'text': quote.css('span.text::text').get(),+ 'author': quote.css('span small::text').get(),'tags': quote.css('div.tags a.tag::text').extract(),}- next_page = response.css('li.next a::attr(href)').extract_first()+ next_page = response.css('li.next a::attr(href)').get()if next_page is not None:yield response.follow(next_page, callback=self.parse)@@ -641,7 +642,7 @@ this time for scraping author information::def parse_author(self, response):def extract_with_css(query):- return response.css(query).extract_first().strip()+ return response.css(query).get().strip()yield {'name': extract_with_css('h3.author-title::text'),@@ -710,11 +711,11 @@ with a specific tag, building the URL based on the argument::def parse(self, response):for quote in response.css('div.quote'):yield {- 'text': quote.css('span.text::text').extract_first(),- 'author': quote.css('small.author::text').extract_first(),+ 'text': quote.css('span.text::text').get(),+ 'author': quote.css('small.author::text').get(),}- next_page = response.css('li.next a::attr(href)').extract_first()+ next_page = response.css('li.next a::attr(href)').get()if next_page is not None:yield response.follow(next_page, self.parse)diff --git a/docs/topics/shell.rst b/docs/topics/shell.rstindex 11ab199f2..9de6abef7 100644--- a/docs/topics/shell.rst+++ b/docs/topics/shell.rst@@ -179,7 +179,7 @@ all start with the ``[s]`` prefix)::After that, we can start playing with the objects::- >>> response.xpath('//title/text()').extract_first()+ >>> response.xpath('//title/text()').get()'Scrapy | A Fast and Powerful Scraping and Web Crawling Framework'>>> fetch("https://reddit.com")diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 697732b47..4505b7315 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -434,8 +434,8 @@ Let's now take a look at an example CrawlSpider with rules::self.logger.info('Hi, this is an item page! %s', response.url)item = scrapy.Item()item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')- item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()- item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()+ item['name'] = response.xpath('//td[@id="item_name"]/text()').get()+ item['description'] = response.xpath('//td[@id="item_description"]/text()').get()return item@@ -548,9 +548,9 @@ These spiders are pretty easy to use, let's have a look at one example::self.logger.info('Hi, this is a <%s> node!: %s', self.itertag, ''.join(node.extract()))item = TestItem()- item['id'] = node.xpath('@id').extract()- item['name'] = node.xpath('name').extract()- item['description'] = node.xpath('description').extract()+ item['id'] = node.xpath('@id').get()+ item['name'] = node.xpath('name').get()+ item['description'] = node.xpath('description').get()return itemBasically what we did up there was to create a spider that downloads a feed from