Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: update data flow description and image (fixes: #2278) This fixes the explanation to use Requests instead of URLs, which is what actually happens, and is also consistent with the new tutorial, which already explains how URLs become Request objects.
Elias Dorneles3ac3ac4d925e
diff --git a/docs/topics/_images/scrapy_architecture_02.png b/docs/topics/_images/scrapy_architecture_02.pngindex 80be02078..5fe393f22 100644Binary files a/docs/topics/_images/scrapy_architecture_02.png and b/docs/topics/_images/scrapy_architecture_02.png differdiff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rstindex 39e54ee99..84c053660 100644--- a/docs/topics/architecture.rst+++ b/docs/topics/architecture.rst@@ -29,20 +29,20 @@ Data flowThe data flow in Scrapy is controlled by the execution engine, and goes likethis:-1. The :ref:`Engine <component-engine>` gets the first URLs to crawl from the+1. The :ref:`Engine <component-engine>` gets the first Requests to crawl from the:ref:`Spider <component-spiders>`.-2. The :ref:`Engine <component-engine>` schedules the URLs in the- :ref:`Scheduler <component-scheduler>` as Requests and asks for the- next URLs to crawl.+2. The :ref:`Engine <component-engine>` schedules the Requests in the+ :ref:`Scheduler <component-scheduler>` and asks for the+ next Requests to crawl.-3. The :ref:`Scheduler <component-scheduler>` returns the next URLs to crawl+3. The :ref:`Scheduler <component-scheduler>` returns the next Requeststo the :ref:`Engine <component-engine>`.-4. The :ref:`Engine <component-engine>` sends the URLs to the+4. The :ref:`Engine <component-engine>` sends the Requests to the:ref:`Downloader <component-downloader>`, passing through the:ref:`Downloader Middleware <component-downloader-middleware>`- (request direction).+ (requests direction).5. Once the page finishes downloading the:ref:`Downloader <component-downloader>` generates a Response (with@@ -62,10 +62,11 @@ this::ref:`Spider Middleware <component-spider-middleware>` (output direction).8. The :ref:`Engine <component-engine>` sends processed items to- :ref:`Item Pipelines <component-pipelines>` and processed Requests to- the :ref:`Scheduler <component-scheduler>`.+ :ref:`Item Pipelines <component-pipelines>`, then sends processed Requests to+ the :ref:`Scheduler <component-scheduler>` and asks for the next+ Requests to crawl.-9. The process repeats (from step 1) until there are no more requests from the+9. The process repeats (from step 2) until there are no more requests from the:ref:`Scheduler <component-scheduler>`.Components@@ -102,7 +103,7 @@ Spiders-------Spiders are custom classes written by Scrapy users to parse responses and-extract items (aka scraped items) from them or additional URLs (requests) to+extract items (aka scraped items) from them or additional requests tofollow. For more information see :ref:`topics-spiders`... _component-pipelines: