Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

docs: update overview spider code to use toscrape.com and minor changes So, this will replace the spider example code from the overview that scrapes questions from StackOverflow by a spider scraping quotes (much like the one in the tutorial), and upates the text around it to be consistent.

Elias Dorneles18bd0b0886f6
diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst
index 6c1291c1f..fb390d79d 100644
--- a/docs/intro/overview.rst
+++ b/docs/intro/overview.rst
@@ -19,74 +19,69 @@ Walk-through of an example spider
In order to show you what Scrapy brings to the table, we'll walk you through an
example of a Scrapy Spider using the simplest way to run a spider.
-So, here's the code for a spider that follows the links to the top
-voted questions on StackOverflow and scrapes some data from each page::
+Here's the code for a spider that scrapes famous quotes from website
+http://quotes.toscrape.com, following the pagination::
import scrapy
- class StackOverflowSpider(scrapy.Spider):
- name = 'stackoverflow'
- start_urls = ['http://stackoverflow.com/questions?sort=votes']
+ class QuotesSpider(scrapy.Spider):
+ name = "quotes"
+ start_urls = [
+ 'http://quotes.toscrape.com/tag/humor/',
+ ]
def parse(self, response):
- for href in response.css('.question-summary h3 a::attr(href)'):
- full_url = response.urljoin(href.extract())
- yield scrapy.Request(full_url, callback=self.parse_question)
+ for quote in response.css('div.quote'):
+ yield {
+ 'text': quote.css('span.text::text').extract_first(),
+ 'author': quote.xpath('span/small/text()').extract_first(),
+ }
- def parse_question(self, response):
- yield {
- 'title': response.css('h1 a::text').extract_first(),
- 'votes': response.css('.question .vote-count-post::text').extract_first(),
- 'body': response.css('.question .post-text').extract_first(),
- 'tags': response.css('.question .post-tag::text').extract(),
- 'link': response.url,
- }
+ next_page = response.css('li.next a::attr("href")').extract_first()
+ if next_page:
+ next_page = response.urljoin(next_page)
+ yield scrapy.Request(next_page, callback=self.parse)
-Put this in a file, name it to something like ``stackoverflow_spider.py``
+Put this in a text file, name it to something like ``quotes_spider.py``
and run the spider using the :command:`runspider` command::
- scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json
+ scrapy runspider quotes_spider.py -o quotes.json
-When this finishes you will have in the ``top-stackoverflow-questions.json`` file
-a list of the most upvoted questions in StackOverflow in JSON format, containing the
-title, link, number of upvotes, a list of the tags and the question content in HTML,
-looking like this (reformatted for easier reading)::
+When this finishes you will have in the ``quotes.json`` file a list of the
+quotes in JSON format, containing text and author, looking like this (reformatted
+here for better readability)::
[{
- "body": "... LONG HTML HERE ...",
- "link": "http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array",
- "tags": ["java", "c++", "performance", "optimization"],
- "title": "Why is processing a sorted array faster than an unsorted array?",
- "votes": "9924"
+ "author": "Jane Austen",
+ "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
},
{
- "body": "... LONG HTML HERE ...",
- "link": "http://stackoverflow.com/questions/1260748/how-do-i-remove-a-git-submodule",
- "tags": ["git", "git-submodules"],
- "title": "How do I remove a Git submodule?",
- "votes": "1764"
+ "author": "Groucho Marx",
+ "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
+ },
+ {
+ "author": "Steve Martin",
+ "text": "\u201cA day without sunshine is like, you know, night.\u201d"
},
...]
-
What just happened?
-------------------
-When you ran the command ``scrapy runspider somefile.py``, Scrapy looked for a
+When you ran the command ``scrapy runspider quotes_spider.py``, Scrapy looked for a
Spider definition inside it and ran it through its crawler engine.
The crawl started by making requests to the URLs defined in the ``start_urls``
-attribute (in this case, only the URL for StackOverflow top questions page)
+attribute (in this case, only the URL for quotes in *humor* category)
and called the default callback method ``parse``, passing the response object as
-an argument. In the ``parse`` callback we extract the links to the
-question pages using a CSS Selector with a custom extension that allows to get
-the value for an attribute. Then we yield a few more requests to be sent,
-registering the method ``parse_question`` as the callback to be called for each
-of them as they finish.
+an argument. In the ``parse`` callback we loop through the quote elements
+using a CSS Selector, yield a Python dict with the extracted quote text and author,
+look for a link to the next page and schedules another request using the same
+``parse`` method as callback.
Here you notice one of the main advantages about Scrapy: requests are
:ref:`scheduled and processed asynchronously <topics-architecture>`. This
@@ -103,10 +98,6 @@ each request, limiting amount of concurrent requests per domain or per IP, and
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
to figure out these automatically.
-Finally, the ``parse_question`` callback scrapes the question data for each
-page yielding a dict, which Scrapy then collects and writes to a JSON file as
-requested in the command line.
-
.. note::
This is using :ref:`feed exports <topics-feed-exports>` to generate the
@@ -145,12 +136,13 @@ scraping easy and efficient, such as:
:ref:`pipelines <topics-item-pipeline>`).
* Wide range of built-in extensions and middlewares for handling:
- * cookies and session handling
- * HTTP features like compression, authentication, caching
- * user-agent spoofing
- * robots.txt
- * crawl depth restriction
- * and more
+
+ - cookies and session handling
+ - HTTP features like compression, authentication, caching
+ - user-agent spoofing
+ - robots.txt
+ - crawl depth restriction
+ - and more
* A :ref:`Telnet console <topics-telnetconsole>` for hooking into a Python
console running inside your Scrapy process, to introspect and debug your
@@ -165,8 +157,8 @@ What's next?
============
The next steps for you are to :ref:`install Scrapy <intro-install>`,
-:ref:`follow through the tutorial <intro-tutorial>` to learn how to organize
-your code in Scrapy projects and `join the community`_. Thanks for your
+:ref:`follow through the tutorial <intro-tutorial>` to learn how to create
+a full-blown Scrapy project and `join the community`_. Thanks for your
interest!
.. _join the community: http://scrapy.org/community/