Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add documentation add section to broad-crawl topic reword in accord with broad-crawl topic add documentation for new priority queue

Vostretsov Nikita338b78d796de
diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst
index eb02086dc..37f7a8748 100644
--- a/docs/topics/broad-crawls.rst
+++ b/docs/topics/broad-crawls.rst
@@ -39,6 +39,17 @@ you need to keep in mind when using Scrapy for doing broad crawls, along with
concrete suggestions of Scrapy settings to tune in order to achieve an
efficient broad crawl.
+Use proper :setting:`SCHEDULER_PRIORITY_QUEUE`
+==============================================
+
+Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``.
+It works best during single domain crawl. And it does not work well with crawling
+many different domains in parallel
+
+To apply recommended priority queue use::
+
+ SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'
+
Increase concurrency
====================
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 47b6cf13d..7b9ff7e39 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -1144,7 +1144,12 @@ SCHEDULER_PRIORITY_QUEUE
------------------------
Default: ``'queuelib.PriorityQueue'``
-Type of priority queue used by scheduler.
+Type of priority queue used by scheduler. Another available type is
+``scrapy.pqueues.DownloaderAwarePriorityQueue``.
+``scrapy.pqueues.DownloaderAwarePriorityQueue`` is works better than
+``'queuelib.PriorityQueue'`` when you crawl many different domains in parallel.
+But ``scrapy.pqueues.DownloaderAwarePriorityQueue``
+does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`.
.. setting:: SPIDER_CONTRACTS