Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Show how to crawl quotes.toscrape.com faster in the tutorial (#8334)

Adrianffd0c465c4c2
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index e6d428acb..522ea6cff 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -136,6 +136,29 @@ and defines some attributes and methods:
the scraped data as dicts and also finding new URLs to
follow and creating new requests (:class:`~scrapy.Request`) from them.
+Crawling faster for this tutorial
+---------------------------------
+
+The :file:`tutorial/settings.py` file of your project makes Scrapy crawl
+politely: it sends at most one request at a time to each website
+(:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`), waiting about one second between
+requests (:setting:`DOWNLOAD_DELAY`). That keeps you from overwhelming
+websites, but it also makes crawling slow, and it means the requests above are
+sent one after another rather than in parallel.
+
+``quotes.toscrape.com`` is a sandbox meant for scraping practice, so we can
+safely crawl it faster. Open :file:`tutorial/settings.py` and add a
+:setting:`DOWNLOAD_SLOTS` entry that raises the concurrency and lowers the
+delay for that domain only:
+
+.. code-block:: python
+
+ DOWNLOAD_SLOTS = {
+ "quotes.toscrape.com": {"concurrency": 16, "delay": 0.1},
+ }
+
+The polite defaults still apply to every other website.
+
How to run our spider
---------------------