Scrapy, a fast high-level web crawling & scraping framework for Python.
broad-crawls.rst: Refactor the memory usage section
Adrián Chavesed487b7db387
diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rstindex 5b9e7832f..1ab08d949 100644--- a/docs/topics/broad-crawls.rst+++ b/docs/topics/broad-crawls.rst@@ -39,6 +39,8 @@ you need to keep in mind when using Scrapy for doing broad crawls, along withconcrete suggestions of Scrapy settings to tune in order to achieve anefficient broad crawl.+.. _broad-crawls-scheduler-priority-queue:+Use the right :setting:`SCHEDULER_PRIORITY_QUEUE`=================================================@@ -50,24 +52,37 @@ To apply the recommended priority queue use::SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'+.. _broad-crawls-concurrency:+Increase concurrency====================Concurrency is the number of requests that are processed in parallel. There is-a global limit and a per-domain limit.+a global limit (:setting:`CONCURRENT_REQUESTS`) and an additional limit that+can be set either per domain (:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`) or per+IP (:setting:`CONCURRENT_REQUESTS_PER_IP`).++.. note:: The scheduler priority queue :ref:`recommended for broad crawls+ <broad-crawls-scheduler-priority-queue>` does not support+ :setting:`CONCURRENT_REQUESTS_PER_IP`.The default global concurrency limit in Scrapy is not suitable for crawlingmany different domains in parallel, so you will want to increase it. How much-to increase it will depend on how much CPU you crawler will have available. A-good starting point is ``100``, but the best way to find out is by doing some-trials and identifying at what concurrency your Scrapy process gets CPU-bounded. For optimum performance, you should pick a concurrency where CPU usage-is at 80-90%.+to increase it will depend on how much CPU and memory you crawler will have+available.-To increase the global concurrency use::+A good starting point is ``100``::CONCURRENT_REQUESTS = 100+But the best way to find out is by doing some trials and identifying at what+concurrency your Scrapy process gets CPU bounded. For optimum performance, you+should pick a concurrency where CPU usage is at 80-90%.++Increasing concurrency also increases memory usage. If memory usage is a+concern, you might need to lower your global concurrency limit accordingly.++Increase Twisted IO thread pool maximum size============================================@@ -175,31 +190,24 @@ and enabling it for focused crawls doesn't make much sense... _ajax crawlable: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started-Reducing memory consumption for broad crawls-============================================+.. _broad-crawls-bfo:-For broad crawls, the amount of memory used for storing `Requests`_, references, and further information may soon become pretty large.-The following steps help to reduce the amount of memory used for broad crawls.+Crawl in BFO order+==================-1) **Change the queue type:** The default queue for crawls is "Last-In-First-Out ( `LIFO`_ )" using the concept of "Depth-First Search ( `DFS`_ )". In case the page scraping is faster than the processing of the spiders, early `Requests`_ might not be processed and therefore block memory until the final depth is reached. Setting the queue from LIFO to "First-In-First-Out ( `FIFO`_ )" and setting dispatching from `DFS`_ to "Breadth-First Search ( `BFS`_ )", as shown in the `FAQ`_ will solve this problem::+:ref:`Scrapy crawls in DFO order by default <faq-bfo-dfo>`.- DEPTH_PRIORITY = 1- SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'- SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'+In broad crawls, however, page crawling tends to be faster than page+processing. As a result, unprocessed early requests stay in memory until the+final depth is reached, which can significantly increase memory usage.-2) **Reduce the number of concurrent requests:** As stated before, the global concurrency level can be set using::-- CONCURRENT_REQUESTS = 100+:ref:`Crawl in BFO order <faq-bfo-dfo>` instead to save memory.-However, if scraping is faster than processing, the queue will eventually exceed the memory size.-Unfortunately, there is yet no autobalancing feature available, so you need to find concurrency values that fit your processing speed.-3) **Use the profiling and trackref capabilities of scrapy:** scrapy provides an own and interactive profiling and reference tracking tool. See `debugging memory leaks`_ for more information.+Be mindful of memory leaks+==========================-.. _debugging memory leaks: http://doc.scrapy.org/en/latest/topics/leaks.html-.. _Requests: http://doc.scrapy.org/en/latest/topics/request-response.html#request-objects-.. _LIFO: http://en.wikipedia.org/wiki/Stack_(abstract_data_type)-.. _FIFO: http://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)-.. _BFS: http://en.wikipedia.org/wiki/Breadth-first_search-.. _DFS: http://en.wikipedia.org/wiki/Depth-first_search-.. _FAQ: http://doc.scrapy.org/en/latest/faq.html#does-scrapy-crawl-in-breadth-first-or-depth-first-order+If your broad crawl shows a high memory usage, in addition to :ref:`crawling in+BFO order <broad-crawls-bfo>` and :ref:`lowering concurrency+<broad-crawls-concurrency>` you should :ref:`debug your memory leaks+<topics-leaks>`.