Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: update overview spider code to use toscrape.com and minor changes So, this will replace the spider example code from the overview that scrapes questions from StackOverflow by a spider scraping quotes (much like the one in the tutorial), and upates the text around it to be consistent.
Elias Dorneles18bd0b0886f6
diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex 6c1291c1f..fb390d79d 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -19,74 +19,69 @@ Walk-through of an example spiderIn order to show you what Scrapy brings to the table, we'll walk you through anexample of a Scrapy Spider using the simplest way to run a spider.-So, here's the code for a spider that follows the links to the top-voted questions on StackOverflow and scrapes some data from each page::+Here's the code for a spider that scrapes famous quotes from website+http://quotes.toscrape.com, following the pagination::import scrapy- class StackOverflowSpider(scrapy.Spider):- name = 'stackoverflow'- start_urls = ['http://stackoverflow.com/questions?sort=votes']+ class QuotesSpider(scrapy.Spider):+ name = "quotes"+ start_urls = [+ 'http://quotes.toscrape.com/tag/humor/',+ ]def parse(self, response):- for href in response.css('.question-summary h3 a::attr(href)'):- full_url = response.urljoin(href.extract())- yield scrapy.Request(full_url, callback=self.parse_question)+ for quote in response.css('div.quote'):+ yield {+ 'text': quote.css('span.text::text').extract_first(),+ 'author': quote.xpath('span/small/text()').extract_first(),+ }- def parse_question(self, response):- yield {- 'title': response.css('h1 a::text').extract_first(),- 'votes': response.css('.question .vote-count-post::text').extract_first(),- 'body': response.css('.question .post-text').extract_first(),- 'tags': response.css('.question .post-tag::text').extract(),- 'link': response.url,- }+ next_page = response.css('li.next a::attr("href")').extract_first()+ if next_page:+ next_page = response.urljoin(next_page)+ yield scrapy.Request(next_page, callback=self.parse)-Put this in a file, name it to something like ``stackoverflow_spider.py``+Put this in a text file, name it to something like ``quotes_spider.py``and run the spider using the :command:`runspider` command::- scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json+ scrapy runspider quotes_spider.py -o quotes.json-When this finishes you will have in the ``top-stackoverflow-questions.json`` file-a list of the most upvoted questions in StackOverflow in JSON format, containing the-title, link, number of upvotes, a list of the tags and the question content in HTML,-looking like this (reformatted for easier reading)::+When this finishes you will have in the ``quotes.json`` file a list of the+quotes in JSON format, containing text and author, looking like this (reformatted+here for better readability)::[{- "body": "... LONG HTML HERE ...",- "link": "http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array",- "tags": ["java", "c++", "performance", "optimization"],- "title": "Why is processing a sorted array faster than an unsorted array?",- "votes": "9924"+ "author": "Jane Austen",+ "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"},{- "body": "... LONG HTML HERE ...",- "link": "http://stackoverflow.com/questions/1260748/how-do-i-remove-a-git-submodule",- "tags": ["git", "git-submodules"],- "title": "How do I remove a Git submodule?",- "votes": "1764"+ "author": "Groucho Marx",+ "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"+ },+ {+ "author": "Steve Martin",+ "text": "\u201cA day without sunshine is like, you know, night.\u201d"},...]-What just happened?--------------------When you ran the command ``scrapy runspider somefile.py``, Scrapy looked for a+When you ran the command ``scrapy runspider quotes_spider.py``, Scrapy looked for aSpider definition inside it and ran it through its crawler engine.The crawl started by making requests to the URLs defined in the ``start_urls``-attribute (in this case, only the URL for StackOverflow top questions page)+attribute (in this case, only the URL for quotes in *humor* category)and called the default callback method ``parse``, passing the response object as-an argument. In the ``parse`` callback we extract the links to the-question pages using a CSS Selector with a custom extension that allows to get-the value for an attribute. Then we yield a few more requests to be sent,-registering the method ``parse_question`` as the callback to be called for each-of them as they finish.+an argument. In the ``parse`` callback we loop through the quote elements+using a CSS Selector, yield a Python dict with the extracted quote text and author,+look for a link to the next page and schedules another request using the same+``parse`` method as callback.Here you notice one of the main advantages about Scrapy: requests are:ref:`scheduled and processed asynchronously <topics-architecture>`. This@@ -103,10 +98,6 @@ each request, limiting amount of concurrent requests per domain or per IP, andeven :ref:`using an auto-throttling extension <topics-autothrottle>` that triesto figure out these automatically.-Finally, the ``parse_question`` callback scrapes the question data for each-page yielding a dict, which Scrapy then collects and writes to a JSON file as-requested in the command line.-.. note::This is using :ref:`feed exports <topics-feed-exports>` to generate the@@ -145,12 +136,13 @@ scraping easy and efficient, such as::ref:`pipelines <topics-item-pipeline>`).* Wide range of built-in extensions and middlewares for handling:- * cookies and session handling- * HTTP features like compression, authentication, caching- * user-agent spoofing- * robots.txt- * crawl depth restriction- * and more++ - cookies and session handling+ - HTTP features like compression, authentication, caching+ - user-agent spoofing+ - robots.txt+ - crawl depth restriction+ - and more* A :ref:`Telnet console <topics-telnetconsole>` for hooking into a Pythonconsole running inside your Scrapy process, to introspect and debug your@@ -165,8 +157,8 @@ What's next?============The next steps for you are to :ref:`install Scrapy <intro-install>`,-:ref:`follow through the tutorial <intro-tutorial>` to learn how to organize-your code in Scrapy projects and `join the community`_. Thanks for your+:ref:`follow through the tutorial <intro-tutorial>` to learn how to create+a full-blown Scrapy project and `join the community`_. Thanks for yourinterest!.. _join the community: http://scrapy.org/community/