Scrapy, a fast high-level web crawling & scraping framework for Python.
Show how to crawl quotes.toscrape.com faster in the tutorial (#8334)
Adrianffd0c465c4c2
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex e6d428acb..522ea6cff 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -136,6 +136,29 @@ and defines some attributes and methods:the scraped data as dicts and also finding new URLs tofollow and creating new requests (:class:`~scrapy.Request`) from them.+Crawling faster for this tutorial+---------------------------------++The :file:`tutorial/settings.py` file of your project makes Scrapy crawl+politely: it sends at most one request at a time to each website+(:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`), waiting about one second between+requests (:setting:`DOWNLOAD_DELAY`). That keeps you from overwhelming+websites, but it also makes crawling slow, and it means the requests above are+sent one after another rather than in parallel.++``quotes.toscrape.com`` is a sandbox meant for scraping practice, so we can+safely crawl it faster. Open :file:`tutorial/settings.py` and add a+:setting:`DOWNLOAD_SLOTS` entry that raises the concurrency and lowers the+delay for that domain only:++.. code-block:: python++ DOWNLOAD_SLOTS = {+ "quotes.toscrape.com": {"concurrency": 16, "delay": 0.1},+ }++The polite defaults still apply to every other website.+How to run our spider---------------------