Scrapy, a fast high-level web crawling & scraping framework for Python.
Add documentation add section to broad-crawl topic reword in accord with broad-crawl topic add documentation for new priority queue
Vostretsov Nikita338b78d796de
diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rstindex eb02086dc..37f7a8748 100644--- a/docs/topics/broad-crawls.rst+++ b/docs/topics/broad-crawls.rst@@ -39,6 +39,17 @@ you need to keep in mind when using Scrapy for doing broad crawls, along withconcrete suggestions of Scrapy settings to tune in order to achieve anefficient broad crawl.+Use proper :setting:`SCHEDULER_PRIORITY_QUEUE`+==============================================++Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``.+It works best during single domain crawl. And it does not work well with crawling+many different domains in parallel++To apply recommended priority queue use::++ SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'+Increase concurrency====================diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 47b6cf13d..7b9ff7e39 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1144,7 +1144,12 @@ SCHEDULER_PRIORITY_QUEUE------------------------Default: ``'queuelib.PriorityQueue'``-Type of priority queue used by scheduler.+Type of priority queue used by scheduler. Another available type is+``scrapy.pqueues.DownloaderAwarePriorityQueue``.+``scrapy.pqueues.DownloaderAwarePriorityQueue`` is works better than+``'queuelib.PriorityQueue'`` when you crawl many different domains in parallel.+But ``scrapy.pqueues.DownloaderAwarePriorityQueue``+does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`... setting:: SPIDER_CONTRACTS