Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: use https scheme for each quotes.toscrape.com url occurrence
Péter Ferenc Gyarmati38d2a154ec79
diff --git a/docs/intro/examples.rst b/docs/intro/examples.rstindex 96363c7d5..edff894c6 100644--- a/docs/intro/examples.rst+++ b/docs/intro/examples.rst@@ -7,7 +7,7 @@ ExamplesThe best way to learn is with examples, and Scrapy is no exception. For thisreason, there is an example Scrapy project named quotesbot_, that you can use toplay and learn more about Scrapy. It contains two spiders for-http://quotes.toscrape.com, one using CSS selectors and another one using XPath+https://quotes.toscrape.com, one using CSS selectors and another one using XPathexpressions.The quotesbot_ project is available at: https://github.com/scrapy/quotesbot.diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex 405bf845d..f3d652621 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -20,7 +20,7 @@ In order to show you what Scrapy brings to the table, we'll walk you through anexample of a Scrapy Spider using the simplest way to run a spider.Here's the code for a spider that scrapes famous quotes from website-http://quotes.toscrape.com, following the pagination::+https://quotes.toscrape.com, following the pagination::import scrapy@@ -28,7 +28,7 @@ http://quotes.toscrape.com, following the pagination::class QuotesSpider(scrapy.Spider):name = 'quotes'start_urls = [- 'http://quotes.toscrape.com/tag/humor/',+ 'https://quotes.toscrape.com/tag/humor/',]def parse(self, response):diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex ca5856881..5697b9608 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -7,7 +7,7 @@ Scrapy TutorialIn this tutorial, we'll assume that Scrapy is already installed on your system.If that's not the case, see :ref:`intro-install`.-We are going to scrape `quotes.toscrape.com <http://quotes.toscrape.com/>`_, a website+We are going to scrape `quotes.toscrape.com <https://quotes.toscrape.com/>`_, a websitethat lists quotes from famous authors.This tutorial will walk you through these tasks:@@ -93,8 +93,8 @@ This is the code for our first Spider. Save it in a file nameddef start_requests(self):urls = [- 'http://quotes.toscrape.com/page/1/',- 'http://quotes.toscrape.com/page/2/',+ 'https://quotes.toscrape.com/page/1/',+ 'https://quotes.toscrape.com/page/2/',]for url in urls:yield scrapy.Request(url=url, callback=self.parse)@@ -143,9 +143,9 @@ similar to this::2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023- 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)- 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)- 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)+ 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://quotes.toscrape.com/robots.txt> (referer: None)+ 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/1/> (referer: None)+ 2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/2/> (referer: None)2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)@@ -184,8 +184,8 @@ for your spider::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'http://quotes.toscrape.com/page/1/',- 'http://quotes.toscrape.com/page/2/',+ 'https://quotes.toscrape.com/page/1/',+ 'https://quotes.toscrape.com/page/2/',]def parse(self, response):@@ -207,7 +207,7 @@ Extracting dataThe best way to learn how to extract data with Scrapy is trying selectorsusing the :ref:`Scrapy shell <topics-shell>`. Run::- scrapy shell 'http://quotes.toscrape.com/page/1/'+ scrapy shell 'https://quotes.toscrape.com/page/1/'.. note::@@ -217,18 +217,18 @@ using the :ref:`Scrapy shell <topics-shell>`. Run::On Windows, use double quotes instead::- scrapy shell "http://quotes.toscrape.com/page/1/"+ scrapy shell "https://quotes.toscrape.com/page/1/"You will see something like::[ ... Scrapy log here ... ]- 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)+ 2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/1/> (referer: None)[s] Available Scrapy objects:[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)[s] crawler <scrapy.crawler.Crawler object at 0x7fa91d888c90>[s] item {}- [s] request <GET http://quotes.toscrape.com/page/1/>- [s] response <200 http://quotes.toscrape.com/page/1/>+ [s] request <GET https://quotes.toscrape.com/page/1/>+ [s] response <200 https://quotes.toscrape.com/page/1/>[s] settings <scrapy.settings.Settings object at 0x7fa91d888c10>[s] spider <DefaultSpider 'default' at 0x7fa91c8af990>[s] Useful shortcuts:@@ -241,7 +241,7 @@ object:.. invisible-code-block: python- response = load_response('http://quotes.toscrape.com/page/1/', 'quotes1.html')+ response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')>>> response.css('title')[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]@@ -355,7 +355,7 @@ Extracting quotes and authorsNow that you know a bit about selection and extraction, let's complete ourspider by writing the code to extract the quotes from the web page.-Each quote in http://quotes.toscrape.com is represented by HTML elements that look+Each quote in https://quotes.toscrape.com is represented by HTML elements that looklike this:.. code-block:: html@@ -379,7 +379,7 @@ like this:Let's open up scrapy shell and play a bit to find out how to extract the datawe want::- $ scrapy shell 'http://quotes.toscrape.com'+ $ scrapy shell 'https://quotes.toscrape.com'We get a list of selectors for the quote HTML elements with:@@ -444,8 +444,8 @@ in the callback, as you can see below::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'http://quotes.toscrape.com/page/1/',- 'http://quotes.toscrape.com/page/2/',+ 'https://quotes.toscrape.com/page/1/',+ 'https://quotes.toscrape.com/page/2/',]def parse(self, response):@@ -458,9 +458,9 @@ in the callback, as you can see below::If you run this spider, it will output the extracted data with the log::- 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>+ 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>{'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}- 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>+ 2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://quotes.toscrape.com/page/1/>{'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}@@ -505,7 +505,7 @@ Following links===============Let's say, instead of just scraping the stuff from the first two pages-from http://quotes.toscrape.com, you want quotes from all the pages in the website.+from https://quotes.toscrape.com, you want quotes from all the pages in the website.Now that you know how to extract data from pages, let's see how to follow linksfrom them.@@ -549,7 +549,7 @@ page, extracting data from it::class QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'http://quotes.toscrape.com/page/1/',+ 'https://quotes.toscrape.com/page/1/',]def parse(self, response):@@ -600,7 +600,7 @@ As a shortcut for creating Request objects you can useclass QuotesSpider(scrapy.Spider):name = "quotes"start_urls = [- 'http://quotes.toscrape.com/page/1/',+ 'https://quotes.toscrape.com/page/1/',]def parse(self, response):@@ -654,7 +654,7 @@ this time for scraping author information::class AuthorSpider(scrapy.Spider):name = 'author'- start_urls = ['http://quotes.toscrape.com/']+ start_urls = ['https://quotes.toscrape.com/']def parse(self, response):author_page_links = response.css('.author + a')@@ -727,7 +727,7 @@ with a specific tag, building the URL based on the argument::name = "quotes"def start_requests(self):- url = 'http://quotes.toscrape.com/'+ url = 'https://quotes.toscrape.com/'tag = getattr(self, 'tag', None)if tag is not None:url = url + 'tag/' + tag@@ -747,7 +747,7 @@ with a specific tag, building the URL based on the argument::If you pass the ``tag=humor`` argument to this spider, you'll notice that itwill only visit URLs from the ``humor`` tag, such as-``http://quotes.toscrape.com/tag/humor``.+``https://quotes.toscrape.com/tag/humor``.You can :ref:`learn more about handling spider arguments here <spiderargs>`.diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex 057b1ec62..96475899f 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -81,18 +81,18 @@ clicking directly on the tag. If we expand the ``span`` tag with the ``class="text"`` we will see the quote-text we clicked on. The `Inspector` lets youcopy XPaths to selected elements. Let's try it out.-First open the Scrapy shell at http://quotes.toscrape.com/ in a terminal:+First open the Scrapy shell at https://quotes.toscrape.com/ in a terminal:.. code-block:: none- $ scrapy shell "http://quotes.toscrape.com/"+ $ scrapy shell "https://quotes.toscrape.com/"Then, back to your web browser, right-click on the ``span`` tag, select``Copy > XPath`` and paste it in the Scrapy shell like so:.. invisible-code-block: python- response = load_response('http://quotes.toscrape.com/', 'quotes.html')+ response = load_response('https://quotes.toscrape.com/', 'quotes.html')>>> response.xpath('/html/body/div/div[2]/div[1]/div[1]/span[1]/text()').getall()['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']@@ -227,7 +227,7 @@ interests us is the one request called ``quotes?page=1`` with thetype ``json``.If we click on this request, we see that the request URL is-``http://quotes.toscrape.com/api/quotes?page=1`` and the response+``https://quotes.toscrape.com/api/quotes?page=1`` and the responseis a JSON-object that contains our quotes. We can also right-clickon the request and open ``Open in new tab`` to get a better overview.@@ -247,7 +247,7 @@ also request each page to get every quote on the site::name = 'quote'allowed_domains = ['quotes.toscrape.com']page = 1- start_urls = ['http://quotes.toscrape.com/api/quotes?page=1']+ start_urls = ['https://quotes.toscrape.com/api/quotes?page=1']def parse(self, response):data = json.loads(response.text)@@ -255,7 +255,7 @@ also request each page to get every quote on the site::yield {"quote": quote["text"]}if data["has_next"]:self.page += 1- url = f"http://quotes.toscrape.com/api/quotes?page={self.page}"+ url = f"https://quotes.toscrape.com/api/quotes?page={self.page}"yield scrapy.Request(url=url, callback=self.parse)This spider starts at the first page of the quotes-API. With each@@ -280,7 +280,7 @@ request::from scrapy import Requestrequest = Request.from_curl(- "curl 'http://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil"+ "curl 'https://quotes.toscrape.com/api/quotes?page=1' -H 'User-Agent: Mozil""la/5.0 (X11; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0' -H 'Acce""pt: */*' -H 'Accept-Language: ca,en-US;q=0.7,en;q=0.3' --compressed -H 'X""-Requested-With: XMLHttpRequest' -H 'Proxy-Authorization: Basic QFRLLTAzM"@@ -304,8 +304,8 @@ daunting and pages can be very complex, but it (mostly) boils downto identifying the correct request and replicating it in your spider... _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools-.. _quotes.toscrape.com: http://quotes.toscrape.com-.. _quotes.toscrape.com/scroll: http://quotes.toscrape.com/scroll-.. _quotes.toscrape.com/api/quotes?page=10: http://quotes.toscrape.com/api/quotes?page=10+.. _quotes.toscrape.com: https://quotes.toscrape.com+.. _quotes.toscrape.com/scroll: https://quotes.toscrape.com/scroll+.. _quotes.toscrape.com/api/quotes?page=10: https://quotes.toscrape.com/api/quotes?page=10.. _has-class-extension: https://parsel.readthedocs.io/en/latest/usage.html#other-xpath-extensionsdiff --git a/docs/topics/logging.rst b/docs/topics/logging.rstindex d593c74c6..3bf23d5f5 100644--- a/docs/topics/logging.rst+++ b/docs/topics/logging.rst@@ -218,7 +218,7 @@ For example, let's say you're scraping a website which returns manyHTTP 404 and 500 responses, and you want to hide all messages like this::2016-12-16 22:00:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring- response <500 http://quotes.toscrape.com/page/1-34/>: HTTP status code+ response <500 https://quotes.toscrape.com/page/1-34/>: HTTP status codeis not handled or not allowedThe first thing to note is a logger name - it is in brackets:diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex f6c95c502..4e105642d 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1597,7 +1597,7 @@ In order to use the reactor installed by Scrapy::def start_requests(self):reactor.callLater(self.timeout, self.stop)- urls = ['http://quotes.toscrape.com/page/1']+ urls = ['https://quotes.toscrape.com/page/1']for url in urls:yield scrapy.Request(url=url, callback=self.parse)@@ -1625,7 +1625,7 @@ which raises :exc:`Exception`, becomes::from twisted.internet import reactorreactor.callLater(self.timeout, self.stop)- urls = ['http://quotes.toscrape.com/page/1']+ urls = ['https://quotes.toscrape.com/page/1']for url in urls:yield scrapy.Request(url=url, callback=self.parse)diff --git a/docs/topics/signals.rst b/docs/topics/signals.rstindex 63ad3a9ad..2fbd0b51c 100644--- a/docs/topics/signals.rst+++ b/docs/topics/signals.rst@@ -60,7 +60,7 @@ Let's take an example::class SignalSpider(scrapy.Spider):name = 'signals'- start_urls = ['http://quotes.toscrape.com/page/1/']+ start_urls = ['https://quotes.toscrape.com/page/1/']@classmethoddef from_crawler(cls, crawler, *args, **kwargs):